The First Token Is Not the Answer: Four Streaming Myths and a Chunk-Timeline Probe

Every chat UI feels fast when the text starts moving. Does that mean the model got faster?

No. It means your client got something to render. I keep watching teams ship "streaming = performance" and then get confused when their p95 stays flat. So let me bust the four myths I hear most, and give you one probe that settles them on your own endpoints.

Myth 1: "Streaming makes responses faster"

It does not. Streaming changes when the same work becomes visible.

The model still has to prefill the prompt and decode every token. The final byte usually arrives at roughly the same time, and sometimes later. Extra framing, chunked transfer encoding, and more round trips all cost a little.

So what does streaming actually buy? Perceived latency and early cancellation. A user can read while generation continues. A client can abort a bad answer at token 20 instead of waiting for token 400. Both are real wins. Neither is throughput.

Ask yourself: are you optimizing completion time, or time-to-first-useful-pixel? Those are different metrics with different fixes.

Myth 2: "Time to first token is the model thinking"

TTFT is a bundle of everything that happened before byte one arrived. Model prefill is only one item in that bundle.

Here is what actually sits inside your TTFT number:

1. DNS lookup and TCP handshake (new connection vs pooled)
2. TLS negotiation, if you are on HTTPS
3. Uploading the request body, including any long prompt
4. Gateway queueing or auth checks
5. Prefill over the prompt tokens
6. Time to emit the first chunk
7. Any proxy or compressor buffering on the way back

Items 1 through 4 are network and infrastructure. Items 5 and 6 are the model. If you report one number called "model latency" and it includes a 300 ms TLS handshake, you are not measuring the model.

Want a cleaner signal? Warm the connection first, then measure. Compare pooled and cold connections side by side. The gap tells you how much of your TTFT was never the model's fault.

Myth 3: "One chunk equals one token"

The wire format does not promise that. A delta can carry one token, several tokens, whitespace only, or nothing at all.

Many OpenAI-compatible servers emit chunks on their own schedule. Some coalesce tokens. Some send role-only deltas first. Some send a final chunk with usage and an empty delta list.

So do not compute tokens per second by counting chunks. You will be wrong by a variable factor. Count decoded characters as a rough proxy, or use usage numbers when the server provides them — and treat their absence as a missing field, not a zero.

Myth 4: "It streams in dev, so it streams in prod"

This one burns people every time. Your laptop talks straight to the endpoint. Production sits behind a proxy that has opinions.

Common buffering suspects:

• Reverse proxies with response buffering enabled by default
• Compression middleware that waits to fill a block before flushing
• Application servers that gzip Server-Sent Events
• Client HTTP libraries that read the whole body before returning

On nginx, proxy_buffering on is the usual culprit; an X-Accel-Buffering: no response header or proxy_buffering off for that location is the usual fix. On the client side, requests.get(..., stream=True) plus iter_content behaves very differently from response.text, which buffers everything. And in Python, plain iter_lines() on text can split on \r characters inside payloads — prefer the async line iterator from httpx.

Test the path you ship. Not the path you develop on.

The probe: a chunk timeline, not a single number

You need three observations per run: time to first chunk, inter-chunk gaps, and total time. Everything else is commentary.

Availability note: if you are using MonkeyCode's free model access and free server option (both operator-supplied availability claims, not benchmarks), the useful move is to run this collector from a fixed region instead of your laptop. Free model access lets you point the same harness at more than one endpoint without a billing surprise. The free server option gives the probe a stable network position, which is the whole point of comparing TTFT numbers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# probe.py — chunk-timeline probe for OpenAI-compatible streaming endpoints # Requires: Python 3.10+, httpx. Not executed here; run it against your own endpoints.  import asyncio, json, os, time import httpx BASE = os.environ["BASE_URL"] # e.g. https://your-endpoint/v1 KEY = os.environ["API_KEY"] MODEL = os.environ["MODEL"] PROMPT = "Count from 1 to 40, one number per line." async def probe(runs: int = 5) -> list[dict]: rows = [] async with httpx.AsyncClient(timeout=60.0) as client: for i in range(runs): t0 = time.perf_counter() first = None last = t0 gaps: list[float] = [] content_deltas = 0 async with client.stream( "POST", f"{BASE}/chat/completions", headers={"Authorization": f"Bearer {KEY}"}, json={"model": MODEL, "stream": True, "messages": [{"role": "user", "content": PROMPT}]}, ) as r: r.raise_for_status() async for line in r.aiter_lines(): if not line.startswith("data:"): continue payload = line[5:].strip() if payload == "[DONE]": break now = time.perf_counter() if first is None: first = now - t0 gaps.append(now - last) last = now delta = json.loads(payload)["choices"][0].get("delta", {}) if delta.get("content"): content_deltas += 1 rows.append({ "run": i, "ttft_s": round(first if first is not None else -1, 3), "total_s": round(time.perf_counter() - t0, 3), "chunks": len(gaps), "content_deltas": content_deltas, "max_gap_ms": round(max(gaps or [0]) * 1000, 1), }) return rows if __name__ == "__main__": for row in asyncio.run(probe()): print(row) 

And a five-second sanity check with curl before you blame the model:

curl -sS -N -D /tmp/headers.txt -o /tmp/stream.txt \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d "{\"model\":\"$MODEL\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Count to 40\"}]}" \ "$BASE_URL/chat/completions" \ -w 'connect=%{time_connect} first_byte=%{time_starttransfer} total=%{time_total}\n' grep -i -E 'content-encoding|x-accel-buffering|transfer-encoding' /tmp/headers.txt 

-N disables curl's own buffering. If first_byte is close to total, you are not streaming. You are downloading.

Reading the timeline

Fill this in per run, then look at the shape rather than the average.

run | ttft_s | total_s | chunks | content_deltas | max_gap_ms ----+--------+---------+--------+----------------+----------- 0 | ? | ? | ? | ? | ? 1 | ? | ? | ? | ? | ? 

Three patterns are worth naming:

• High TTFT, low max gap. Cold connection, long prompt, or queueing. Fix the caller or the queue first.
• Low TTFT, one huge gap. Something stalls mid-stream. Look at proxy buffers, timeouts, and upstream retries.
• Low TTFT, many chunks, total close to TTFT. Generation is trivially short. There is nothing to optimize here.

Notice what is missing from that list: "the model is slow." That conclusion needs evidence, not vibes.

Stream or not? A decision table

Limitations and who should skip this

This probe measures your path, not the vendor's internals. Your numbers are not a benchmark of anyone else's service, and they will move when your region, proxy, or prompt length changes.

The script above is a harness, not a result. I have not run it against your endpoints, so treat every field as something you must fill in yourself. It also assumes OpenAI-compatible SSE framing; other protocols need their own parser.

Skip this if you have no request IDs in your logs — you cannot correlate a slow run with anything. Skip it if your traffic is all batch. Skip it if you only want a number for a slide and not a fix.

Run the probe five times before you touch a config. Run it again after every proxy change. Then ask the only question that matters: did the first chunk arrive sooner for a reason you can point at?