Is LiteLLM Rust Really 7x Faster? A Closer Look at the Numbers

Benchmark headlines are easy to share and hard to check.

In July, LiteLLM published benchmarks for its new Rust AI gateway. The headline numbers were big: roughly 7x lower overhead and 9x less memory than Bifrost, plus about 4.5x more requests per dollar. For anyone choosing an LLM gateway, numbers like that make the decision sound easy.

The Bifrost team then went through LiteLLM's raw CSV and JSON files and published a response: Debunking LiteLLM Rust Benchmarking, Part 1. Their conclusion is that the raw data tells a much less dramatic story than the headlines.

I've written before about why I switched from LiteLLM to Bifrost for production systems, so I was curious. In this article, I'll walk through what LiteLLM claimed, what the raw numbers actually show, where each side has a fair point, and how you can check the results yourself.

I'll cite the numbers as published and point out the caveats on both sides.

What LiteLLM Claimed About Its Rust Gateway

LiteLLM's benchmark post, published in July 2026, compared four gateways: the LiteLLM Rust beta, LiteLLM Python v1, Bifrost v1.6.4, and Portkey OSS. All of them pointed at the same local mock provider, so the numbers reflect each gateway's own overhead rather than real model latency.

The headline results looked like this:

From those, LiteLLM's site and blog drew four big claims:

• ~7x lower overhead than Bifrost
• ~9x less memory than Bifrost
• 3.5x lower overhead than "the next AI gateway"
• ~4.5x more requests per dollar than Bifrost

To be fair to LiteLLM, its own post includes some caveats. It says the benchmark is vendor-run, that streaming and the full feature set are still landing, and that readers should treat the differences as order-of-magnitude.

What Bifrost's Team Found in the Raw Data

Instead of running a new benchmark first, the Bifrost team went back to LiteLLM's own published files in the ai-gateway-bench repo and checked whether the headlines matched the data. Here are the main issues they raised.

1. The "7x" overhead number doesn't appear in the sweep

The 4.5 ms Bifrost overhead figure used for the "7x" claim doesn't show up in any row of LiteLLM's latency sweep. In the sweep itself, Bifrost and LiteLLM Rust trade places depending on concurrency (more on that below).

2. The "3.5x" claim is against Portkey, not Bifrost

The homepage line about "3.5x lower overhead than the next AI gateway" compares against Portkey (2.29 ms vs 0.66 ms). Easy to miss, and easy to misread as a Bifrost comparison.

3. "Memory at rest" is actually peak memory

The chart is labeled "memory at rest" and "idle," but the underlying column is peak_rss_mb. LiteLLM's files also contain three different memory pairs, and the headline uses the widest one (199.1 MB vs 21.8 MB). The other pairs are 144 MB vs 26.1 MB, and 282 to 298 MB vs 35 to 36 MB.

4. The configs weren't like-for-like

LiteLLM says logging, spend tracking, and persistence were disabled for every gateway. But according to the Bifrost team's reading, the Bifrost config in the repo doesn't actually set enable_logging to false, and Bifrost's logging plugin is enabled by default in gateway mode. So Bifrost was carrying logging overhead (including SQLite memory) that the Rust gateway wasn't. The Bifrost team notes this reading is based on newer code and not yet confirmed on v1.6.4.

5. An old Bifrost version

The benchmark used Bifrost v1.6.4. Since then, Bifrost has shipped v1.6.5 to v1.6.11 and v2.0.0 to v2.4.0, and the Bifrost changelog now lists v2.5.1. The Bifrost team says 2.x substantially reduced memory use, though this post doesn't measure that yet.

6. Missing setup details and single runs

The hardware, core counts, worker counts, and mock latency aren't published, so "identical hardware" can't be verified. Each result also appears to be a single run with no error bars, and the Bifrost team says rankings flip between files.

7. A partial Rust port

The Bifrost post points out that LiteLLM's Rust migration is still in progress. By source bytes, the repo is 3.11% Rust, and LiteLLM's own tracker reports 24% of major APIs built as of v1.107.0-rc.2. In the Bifrost team's words, the benchmark measures "a narrower Rust beta, not the full LiteLLM product rewritten in Rust."

Smaller issues

• The throughput table shows a dash for Bifrost, which hides a gap of only 0.1% to 2.5%.
• A footnote says "single client," but the sweep runs from 1 to 64 concurrent requests.
• A "~21% CPU" figure doesn't appear in any Rust CPU file; the highest is 15.9%.
• A concurrency-256 run exists in the results folder but was left out of the sweep.

The Numbers: What LiteLLM's Own Data Shows

These figures come straight from LiteLLM's own results/latency_vs_concurrency.csv in the ai-gateway-bench repo. I checked every value below against it. They match the figures in the Bifrost post, and every run had zero errors.

Added p99 latency over a direct call (ms)

The two trade places: Bifrost is lower at concurrency 1 and 64, LiteLLM Rust at 4 and 16. At concurrency 4, the gap is just 0.02 ms. That is a long way from "7x."

Throughput (requests per second)

At the highest tested concurrency, Rust and Bifrost are separated by about 0.1%. For practical purposes, that's a tie.

Memory, CPU, and Concurrency 256: What the Files Show

One caveat before the numbers: they come from a setup where Bifrost was likely running with logging on and Rust wasn’t, on a Bifrost version that’s several releases old.

The Bifrost post reports these results openly, and I think they’re worth repeating:

• Memory and CPU. Rust uses less in every file. Its highest memory reading (~36 MB) is below Bifrost's lowest (~144 MB). Mean CPU was 0.66% for Rust vs 2.59% for Bifrost.
• Concurrency 256. In this single exploratory run, Rust had lower p50 (55.5 ms vs 89.3 ms) and lower p99 (2,667.8 ms vs 5,421.3 ms). Even the direct baseline went past the benchmark's threshold here, so the Bifrost team treats it as exploratory, not a capacity result.

The "requests per dollar" figure is a model

The 4.5x requests-per-dollar figure and the cost per 1M requests are calculated, not measured end to end. They're based on an assumed 4 vCPU / 16 GB instance and the CPU and memory numbers above. So if the memory comparison isn't like-for-like, the cost comparison inherits the same problem.

How to Run the LiteLLM Rust Benchmark Yourself

The best way to settle a benchmark argument is to run it on your own hardware. LiteLLM's benchmark lives in the BerriAI/ai-gateway-bench repo, and its blog outlines the basic flow:

# 1. Clone the repo and set up the environment git clone https://github.com/BerriAI/ai-gateway-bench.git cd ai-gateway-bench python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # 2. Start the mock upstream provider cargo run --release -p mock-upstream # 3. Start each gateway pointed at the mock # (follow gateways/<name>/README.md for each one) # 4. Run a scenario, then regenerate the charts python analyze/make_chart.py 

If you run it, these are the changes that make the comparison fair, based on the gaps the Bifrost team found:

1. Use a current Bifrost build, not v1.6.4. The Bifrost setup docs show how to start the latest version with NPX or Docker.
2. Match the configs. Turn logging off on every gateway, or turn it on for all of them. Don't compare one with logging against one without.
3. Run each scenario several times and report the spread, not a single run.
4. Write down your setup: machine type, CPU cores, worker counts, mock latency, and concurrency for every run.
5. Label memory honestly. Report idle memory and peak memory separately.
6. Keep every concurrency level you tested, including the ones where results get messy.

That's also exactly what the Bifrost team says it will do in Part 2: same benchmarks, current build, logging on and off, repeated trials, and full hardware disclosure.

How to Read Any LLM Gateway Benchmark

This back-and-forth is a useful reminder that a benchmark is only as good as its setup. Whatever gateway you're evaluating, these are the questions I'd ask before trusting a chart:

• Is the raw data published? If you can't see the CSVs, you can't check the headline.
• Do the headline numbers appear in the data? A "7x" claim should trace back to a specific row.
• Were the configs equivalent? Logging, caching, plugins, and persistence all change results.
• Which versions were tested? Gateways move fast, and an old build can be months behind.
• What hardware was used? "Identical hardware" means little without the machine type.
• How many runs? One run per scenario can't tell signal from noise.
• Is it measured or modeled? Cost-per-request figures are often calculations built on other numbers.
• Who ran it? Vendor benchmarks, including Bifrost's own, deserve the same scrutiny.

And most importantly: benchmark the features you'll actually use. A gateway with logging, budgets, fallbacks, and caching turned on behaves very differently from a stripped-down proxy forwarding requests to a mock.

Final Thoughts

LiteLLM's Rust gateway is a real step forward from LiteLLM Python. The jump from hundreds of milliseconds of added latency to under a millisecond is impressive, and its memory and CPU footprint is genuinely small.

But the headline comparisons against Bifrost don't hold up well against LiteLLM's own raw data. On latency, the two trade places depending on concurrency. On throughput, they're within about 0.1% at the top tested load. The bigger memory and cost gaps come from a setup that compared an old Bifrost version, likely with logging on, against a Rust beta with logging off.

So instead of "7x" and "9x," the honest summary is closer to this: on the tested setup, Bifrost and LiteLLM Rust perform about the same on latency and throughput, and Rust uses less memory and CPU in a configuration that wasn't like-for-like.

I'm looking forward to Part 2, where the Bifrost team plans to rerun everything with a current build and full disclosure. Until then, the best advice is the simplest one: don't pick a gateway from a headline. Run the benchmark on your own workload, with the features you'll actually use.

If you want to try Bifrost yourself, it's open source and you can find it at getmaxim.ai/bifrost.