If you have ever picked a model because it "felt faster", you have probably been reading noise.

Across four measurement windows against the same API, one pattern kept showing up: the spread inside three consecutive calls is larger than the difference between most adjacent models in a latency leaderboard.

Below is the fourth window, measured 2026-11-22 03:01 UTC. Three consecutive samples per model, about 1.5 s apart, stream: true, max_tokens: 16, prompt "Reply with exactly: OK". The number is time to first token, in seconds.

model samples median spread
deepseek-v4-flash 0.237 / 0.198 / 0.322 0.237 1.6x
deepseek-v4-pro 0.254 / 0.251 / 0.360 0.254 1.4x
qwen3.8-max 3.068 / 1.766 / 1.187 1.766 2.6x
minimax-m3 1.363 / 2.556 / 1.977 1.977 1.9x
kimi-k2.8-preview 3.735 / 4.789 / 1.671 3.735 2.9x
glm-4.7-flash 429 / 429 / 429 - -

Three findings, in order of how much they should change your reading.

1. The within-model spread beats the between-model difference

qwen3.8-max and minimax-m3 have medians of 1.77 s and 1.98 s, a 12% difference, and internal spreads of 2.6x and 1.9x. "Which of these two is faster" is not answerable from this data. The noise inside one model is an order of magnitude larger than the gap between the two.

kimi-k2.8-preview is the clearest case: 1.671 s to 4.789 s across three calls placed one second apart. A single probe of that model could honestly report "fast" or "slow" and be wrong about half the time.

A TTFT figure therefore needs at least three samples plus a reported median and range. One number is not a measurement, it is a draw from a distribution.

2. A transient looks exactly like a trend

Earlier windows for deepseek-v4-pro, same probe:

timepoint median TTFT
T1 0.20 s
T2 4.99 s
T3 4.39 s
T4 0.254 s

Two consecutive windows showed roughly a 22x degradation, and then it recovered fully.

I reported T2 and T3 as a 22x swing. With T4 in hand the honest description is "a transient episode that recovered". Anyone reading only T2 and T3 would have concluded the model was persistently degraded, and would have been wrong.

That generalises: one bad reading is not evidence of unreliability, and two consecutive bad readings are still not a trend if you stop measuring after them.

3. A capacity signal can hide inside a latency table

glm-4.7-flash returned HTTP 429 on all three probes, one and a half seconds apart, with no other traffic from the same key. That is not a latency number. It is a capacity or quota finding wearing the costume of a missing row. It also happened to be one of the five models the feed lists as free, and a free model is the one most likely to be saturated.

What to do with this

  • Report the median, the range, and n. A number without all three is a rumour.
  • Do not rank the middle of a latency leaderboard. The middle is inside the noise.
  • Keep the timestamp in the artefact. "Fast" with no measurement window does not survive a week.
  • Treat a missing row as data. A 429 is a finding, not a gap.

The reproduction is four lines:

POST /v1/chat/completions {"model": "<id>", "stream": true, "max_tokens": 16, "messages": [{"role": "user", "content": "Reply with exactly: OK"}]} 

Measure the time to the first SSE chunk that is not [DONE]. Repeat at least three times. Report the median and the range.