TL;DR — Constrained decoding for structured output (JSON mode, grammar-based function calling) doesn't just check a model's answer — it rewrites the model's token-by-token probability distribution in real time. That reshaping can force a model into low-probability paths it never 'wanted' to take, and once it's off-distribution, errors compound through the rest of the generation. Schema design is a sampling intervention, not a validation layer, and treating it as free output hygiene causes silent correctness regressions.
Every team that adopts function calling or JSON-schema-constrained output tells itself the same story: we added a validation layer. The model generates, the schema checks, malformed output gets rejected or retried. Clean separation of concerns.
That story is wrong for the most common implementation of structured output: grammar-constrained decoding. It isn't validation. It's a real-time intervention in the model's sampling process, and it changes what the model says, not just whether what it says is well-formed. Most teams have never looked closely enough at the mechanism to notice.
What constrained decoding actually does
When a model generates text under a JSON schema or a grammar (this is how most "guaranteed valid JSON" and typed function-calling implementations work, whether via grammar libraries, FSM-based token maskers, or vendor-side constrained sampling), the decoder doesn't let the model choose freely at each step. At every token position, it computes the full probability distribution over the vocabulary, then masks out every token that would violate the grammar, renormalizes, and samples from what's left.
This guarantees syntactic validity. It says nothing about whether the token the model actually wanted to produce was in the surviving set. If the model's top five candidates at a given position are all grammar-invalid — wrong type, wrong key, a natural-language hedge like "approximately" where the schema demands a bare integer — the decoder throws all of them away and samples from the tail of the distribution. You get a syntactically perfect value that the model assigned, say, 0.3% probability to.
That's the part the "it's just validation" mental model misses. Masking logits is not filtering bad outputs after the fact. It's forcing the model down a path its own probability mass didn't support, at every single constrained step, for the entire generation.
Schema order forces commitment order
Language models generate left to right. A JSON schema with keys in a fixed order isn't just describing data — it's dictating the order in which the model has to commit to answers. If your schema puts "answer" before "reasoning" or "confidence", you are forcing the model to commit to a final answer before it has "seen" any of the explanation it would otherwise generate to arrive at that answer.
This is the opposite of chain-of-thought. Unconstrained, a capable model often reasons its way toward an answer through intermediate tokens — that's the actual mechanism by which extended reasoning improves accuracy. A rigid schema that front-loads the answer field short-circuits that mechanism entirely. You're not getting a reasoning model with structured output bolted on. You're getting a model forced to guess first and rationalize second, inside a format that makes the guess look authoritative because it's cleanly typed.
Field order in a schema is a prompt-engineering decision disguised as a data-modeling decision. Most teams design schemas around what's convenient for the downstream consumer — the API contract, the database column order — and never consider that the order also controls the model's internal generation sequence.
Compounding drift inside a single generation
The damage doesn't stay local to one forced token. Autoregressive generation conditions every future token on everything generated so far, including the tokens the grammar forced. If the decoder pushes the model into a low-probability value at token 40, tokens 41 through 200 are now all conditioned on a context the model never would have produced on its own. The model isn't resuming its original train of thought — it's rationalizing around a commitment it didn't choose.
This is where structured-output failures stop looking like formatting bugs and start looking like hallucination. A model forced into a wrong enum value at the start of a response will frequently generate a plausible-sounding justification for that wrong value in a trailing explanation field, because by the time it generates the explanation, the "wrong" value is just ground truth in its own context window. The schema made the error, and the model's own autoregressive nature then builds a coherent-sounding case for it. Reviewing the output in isolation, it looks confident and internally consistent. It is — consistent with a branch the model was pushed into, not with the branch it would have chosen.
Grammar constraints and fine-tuned JSON mode are different failure modes
It matters which mechanism is producing your "structured output," because the failure shapes differ. Hard grammar constraints (FSM/CFG-based token masking) guarantee syntax but can force arbitrarily low-probability content, as above. Fine-tuned "JSON mode," where a model has been trained to natively prefer valid JSON without external masking, has the opposite failure profile: it's softer on syntax — it can still emit malformed output under distribution shift — but it degrades gracefully in content, because nothing is forcing the model off its own preferred distribution. It's drifting on its own terms, not being shoved.
Many production stacks quietly mix both: a fine-tuned base behavior plus a hard grammar as a backstop. That's reasonable, but it means your failure analysis has to ask which layer produced a given bad field — a model preference problem or a decoder-forced one — and most logging setups don't capture that distinction at all. "The JSON was valid but wrong" is being treated as one bucket when it's at least two different root causes with different fixes.
Designing schemas for the sampler, not just the consumer
If structured output is a sampling intervention, schema design needs to be done with sampling in mind, not just downstream data contracts in mind.
Put free-form reasoning or evidence fields before conclusion fields, so the model's own generative process produces the conclusion instead of being forced to assert it upfront.
Prefer a two-pass pattern for anything non-trivial: unconstrained generation for reasoning, then a separate constrained pass (or a cheap extraction model) that maps the free text into the typed schema. This decouples "let the model think" from "make the output typed," instead of trying to do both in one constrained generation.
Treat enums and tightly typed fields as high-risk. The narrower the valid set, the more likely the decoder has to reach into low-probability territory when the model's actual preference falls outside it. Loosen constraints where you can tolerate a validation step after the fact instead of a hard mask during generation.
Log the unmasked top-k distribution at constrained positions in evaluation, not just production. If the model's top candidate before masking regularly disagrees with what the grammar allowed, that's a signal your schema is fighting the model rather than channeling it.
Function calling and typed interfaces made LLM output programmable, and that's a real engineering win worth keeping. But the mechanism that makes it possible — token-level logit masking — is not a passive filter sitting outside the model's reasoning. It's inside it, shaping every token the moment the schema has an opinion. Treat schema design like you'd treat any other intervention in a probabilistic system: something with a cost, a failure mode, and a place in your evaluation suite — not a type checker you bolt on and forget.
