temperature=0 Isn't Deterministic: 17 of 5,000 Greedy Tokens Flipped

I had a regression test that sent the same prompt to an LLM endpoint with temperature=0 and diffed the output against a saved golden file. It passed on Monday. It failed on Tuesday. Same prompt, same model, same parameters. The two outputs matched word for word for about 140 tokens, then one said "however" where the other said "but", and from there they were two different essays.

I assumed the provider had swapped the model. They hadn't. The actual reason: temperature=0 isn't deterministic, and it's not a bug in anyone's sampler. It's floating-point arithmetic plus the fact that your request shares a GPU with strangers.

TL;DR

• temperature=0 means greedy decoding: pick the argmax token every step. Picking an argmax is deterministic, but computing the logits it's picked from is not.
• Floating-point addition isn't associative. In float16, (1024 + 0.5) + 0.5 gives 1024, while 1024 + (0.5 + 0.5) gives 1025.
• GPU kernels split matrix multiplications into chunks, and the chunking depends on batch size and shape. In an inference server, batch size depends on how much traffic other users send at that moment.
• In my toy float16 simulation, changing only the chunk size changed 67% of the logits and flipped the top token for 17 of 5,000 vectors (0.34%). Over a 500-token generation, that rate adds up to roughly an 82% chance of at least one flip.
• One flipped token forks the whole rest of the output. Fix it with batch-invariant kernels if you control the server. If you don't, stop testing for exact string equality.

Why isn't temperature=0 deterministic?

Because the logits shift slightly from run to run, and greedy decoding is only deterministic when its inputs are. Temperature scales the logits before sampling. At zero, every serious library swaps in a plain argmax. That part is fine.

The problem is upstream. Every logit is a dot product: a hidden vector of a few thousand numbers multiplied element-wise by a row of the output projection and summed. Summing a few thousand floats in a different order gives a slightly different answer. When the top two candidates are close, "slightly different" is enough to change which one wins.

Here's the smallest demo of that I know:

import numpy as np a, b, c = np.float16(1024.0), np.float16(0.5), np.float16(0.5) print((a + b) + c) # 1024.0 print(a + (b + c)) # 1025.0 

At 1024, float16 can only represent steps of 1.0. Adding 0.5 rounds back to 1024 (round half to even), twice. Adding 1.0 at once survives. Same three numbers, different order, different result. Scale that up to 4,096-element dot products across a 100K-token vocabulary and you get logits that move in the third decimal place depending on how the sum was grouped.

What actually changes the summation order on a GPU?

The kernel's tiling strategy changes it, and the strategy depends on the shape of the work. A matrix multiply doesn't add 4,096 products in one line. It splits the reduction dimension into tiles, computes partial sums in parallel, then combines them. Libraries like cuBLAS pick different tile sizes and "split-K" strategies depending on matrix dimensions, because the fastest kernel for a batch of 1 isn't the fastest for a batch of 64.

So the summation order for your request depends on:

1. Batch size. Inference servers with continuous batching (vLLM, TGI, SGLang and every hosted API built on similar ideas) pack whatever requests arrive together into one forward pass. Your batch size is a function of other people's traffic.
2. Sequence length and padding. Different shapes can route to different kernels.
3. Attention kernel choices. Paged or split attention can reduce over the KV cache in chunks, and chunk boundaries move as the cache grows.
4. Hardware. A different GPU generation usually means a different kernel.

Each forward pass is deterministic in isolation. Run the same batch on the same machine twice and you'll usually get identical bits. What isn't fixed is which batch your request lands in. That's the key insight, and it's why "the GPU is nondeterministic" is the wrong mental model. The real issue is batch invariance: your output depends on who you were batched with.

How often does reduction order flip the argmax?

I wanted a number instead of a vibe, so I wrote a toy. It's not a real model. It's random float16 weights for an 8,000-token "vocabulary" with a 4,096-dimension hidden state, and the only variable is how the dot product is chunked: partial sums of 256 elements versus 1,024 elements, each accumulated into a float16 running total.

import numpy as np rng = np.random.default_rng(0) V, D, N = 8000, 4096, 5000 W = (rng.standard_normal((V, D)) / np.sqrt(D)).astype(np.float16) H = rng.standard_normal((N, D)).astype(np.float16) def logits(h, chunk): acc = np.zeros(V, dtype=np.float16) for s in range(0, D, chunk): part = W[:, s:s+chunk].astype(np.float32) @ h[s:s+chunk].astype(np.float32) acc = (acc + part.astype(np.float16)).astype(np.float16) return acc flips = sum(logits(h, 256).argmax() != logits(h, 1024).argmax() for h in H) print(flips, "of", N) 

Results from that run:

0.34% sounds harmless until you remember generation is sequential. If each step independently had a 0.34% chance of flipping, a 500-token answer would have 1 - (1 - 0.0034)^500 ≈ 0.82, so an 82% chance of at least one divergence.

Two big caveats. Positions aren't independent in a real model, and real logit distributions aren't Gaussian. Real models often have genuine near-ties ("but" versus "however", a comma versus a period), which pushes the flip rate up at exactly the positions where language is ambiguous. Confident positions, like the closing bracket in code, almost never flip. Treat my 82% as an illustration of compounding, not a measurement of any production model.

Why does one flipped token change the whole answer?

Because every later token is conditioned on every earlier one. Once position 140 says "however" instead of "but", position 141 sees a different prefix, gets different hidden states, and the logits are no longer the same distribution at all. It's not a small error that persists. It's a fork.

That's why nondeterministic outputs look the way mine did: identical for a stretch, then fully different. If you ever diff two temperature=0 outputs and see a shared prefix that suddenly splits, you're looking at a single argmax flip, not a model swap.

Does setting a seed fix temperature=0 nondeterminism?

No. A seed controls the random number generator used for sampling. Greedy decoding never touches the random number generator, so a seed has nothing to fix. The variance comes from arithmetic, not randomness.

The same goes for torch.use_deterministic_algorithms(True) and CUBLAS_WORKSPACE_CONFIG=:4096:8. Those make a given operation on a given shape reproducible. They don't make a batch of 1 and a batch of 37 produce identical bits for your row. You still need the shapes to match, or kernels that don't care about the batch dimension.

How do I get deterministic LLM output in production?

It depends on whether you own the inference server.

If you run the model yourself:

• Use batch-invariant kernels. Thinking Machines published a well-known write-up in 2025 showing that the main culprits are the reduction kernels (RMSNorm, matmul and attention) and that fixing their reduction strategy across batch sizes yields bitwise-identical outputs. Inference engines have since started shipping batch-invariant modes. Expect a throughput cost, because you're giving up the fastest kernel for each shape.
• For tests only, run at batch size 1 with fixed max lengths. Slow, but stable on one machine.
• Pin the GPU type and library versions in anything that compares outputs across days.

If you call a hosted API:

• Assume temperature=0 gives you low variance, not zero variance. Providers generally document it that way.
• Cache responses keyed by the full request when you need replayability. A cache is the only determinism you fully control.
• Replace exact-match golden tests with structural checks: parse the JSON, assert on fields, compare embeddings or run a rubric. My regression test now checks that the answer contains the three facts it must contain, not that it's byte-identical to Monday.
• For evals, run each case several times and report the spread. A 1-point benchmark change from a single temperature=0 run can be pure reduction-order noise.

The short answer

temperature=0 isn't deterministic because greedy decoding takes the argmax of logits that are computed with floating-point sums, and the order of those sums depends on how the GPU kernel tiles the work, which depends on batch size, which depends on other traffic hitting the same server. Floating-point addition isn't associative, so different groupings produce slightly different logits, and when the top two tokens are nearly tied the winner flips. One flip forks every token after it. A seed doesn't help, since greedy decoding uses no randomness. Batch-invariant kernels fix it if you own the server. If you don't, cache the outputs you need to replay and test for meaning rather than exact strings.

Written by the developer behind Preterview, an interview prep platform.