Structured Output Reliability at Scale: When JSON Schema Validity Breaks Down Under Concurrency

Every replayable measurement here comes from a published script. truncation_experiment.py produces the cut-point and validation-ladder numbers, backend_conformance_probe.py the keyword-enforcement grid, validity_ramp_harness_v4.py the sustained, streaming, multi-turn, follow-up and cardinality ramps, and plot_figures.py the figures. The scripts and retained outputs are in the GitHub repository. The disclosed 37% pre-fix pilot is the one exception, because its aggregate result was not retained. Two arms predate the consolidation and ship as their own files: validity_ramp_harness_burst.py for the bounded-burst ramp, and validity_ramp_harness_managed.py for the managed endpoint. Both are superseded for new work, and neither records TTFT, server metrics or per-request outputs. That’s why those columns are absent from their arms.

Everything measured below hangs on five categories. Naming them is what lets you say “that’s a constraint-boundary failure, not truncation” about your own incident, instead of filing it under structured output “sometimes doesn’t work.” DO’s existing tool-calling article established the single-request foundation this taxonomy stress-tests: a well-defined output schema is “the most effective way to prevent the LLM from hallucinating data points.” Everything below is what happens to that guidance once token budgets and multiple grammar backends enter the picture.

1. Truncation failures. The decoder runs out of token budget before the document closes. A finish_reason: length (or the Responses/Anthropic equivalent) mid-document means everything up to the cut was grammatically valid and the whole thing is unparseable. That’s true by construction, since every proper prefix of a top-level JSON object is missing its closing brace. Reasoning models make this worse in a specific way: thinking tokens draw from the same budget, so a request can spend its entire allowance reasoning, emit zero JSON, and return null content before the model ever reaches the schema.
2. Constraint-boundary failures. The backend accepts a schema and doesn’t fully enforce it, so “strict” quietly isn’t. This is provider- and version-specific. A keyword rejected outright by one backend is silently accepted-but-ignored by another, and the same keyword can be enforced or ignored on the identical backend depending on whether the schema fragment carries a sibling "type".
3. Semantic failures inside valid syntax. Required fields are present, correctly typed, and holding the wrong value. An invoice total that doesn’t match its line items is schema-valid and business-rule-wrong at the same time, because a number is still a number whether or not it’s the right one. No decoding constraint can reach this, by design. Schema validity is a floor, not a ceiling.
4. Extraction and parser failures. When tool-call arguments aren’t schema-constrained, either because tool_choice is "auto" without strict: true or because a provider never offered constrained tool arguments at all, extraction falls back to a model-specific text parser. Malformed markers, multiple calls in one turn, and streaming-chunk reassembly are where that breaks. The misattribution runs both ways: well-formed output a parser mangles gets blamed on the model, and genuinely malformed output the model produced gets blamed on the parser.
5. Contention-induced failures. Grammar compilation is per-schema work with a size-bounded cache, and concurrent requests carrying many distinct schemas compete for it, alongside ordinary scheduler preemption and memory pressure under load. Whether validity itself degrades under concurrency, or only latency does, is treated below as a hypothesis the experiment tests rather than an assumption it confirms.

The measurements turn out to justify two further notes:

• Transport-level failures, meaning a connection the server accepts and drops or a read that dies mid-response, get a bucket of their own outside these five, on purpose. They produce no document at all, so they’re easy to miscount as malformed output, and filing them under contention would have been the exact misattribution this taxonomy exists to prevent.
• A concurrency number alone doesn’t establish that contention occurred, because offered load below the server’s batch capacity can’t queue. Check the level against max_num_seqs before its result means anything.

The experiment and results measure all five on real infrastructure: constraint-boundary and semantic failures directly, truncation both synthetically and at a tightened budget, contention as queueing past the batch capacity and as grammar-cache pressure, and extraction/parser failures as the markdown-fenced responses that sink a third of the unenforced baseline. Contention comes back a null on validity and a large effect on latency and throughput, and the transport failures that first looked like its evidence turn out to belong to the client. What remains sourced from vLLM’s code rather than measured is the tool-call argument-extraction path, where a model-specific text parser rather than a grammar decides what your application sees. The retry design under Building for failure is where that turns into something you can act on.

Constrained decoding masks the sampling distribution. At each step the engine computes which tokens could legally come next given the grammar and the tokens so far, then zeroes out everything else before sampling. XGrammar (Dong, Ruan, Cai, Lai, Xu, Zhao, Chen, MLSys 2025) does this with a byte-level pushdown automaton. It splits the vocabulary into context-independent tokens, which are validatable from the stack top alone and precomputable into a mask cache, and context-dependent tokens, which need the whole stack. For Llama-3.1 with a JSON grammar, only 1,134 of 128,000 tokens are context-dependent, which is why it’s cheap; the paper’s Table 2 shows time-per-output-token for JSON Schema going 6.2 ms to 6.3 ms at batch size 1.

Outlines (Willard & Louf, arXiv:2307.09702, 2023) precomputes an index from FSM states to allowed token sets instead, so the per-token lookup costs “O(1) on average.” Read that claim narrowly. It’s a preprint with one uncaptioned timing plot against a single baseline, one sample per data point, and no index-build measurements at all. The O(1) covers the mask-construction step only, and the index-build cost is declared “effectively irrelevant” by assumption. That assumption matters later.

Either way the invariant is a prefix invariant. After every token, the string so far is a valid prefix of some document matching your grammar. Nothing says the model will reach the end.

OpenAI’s launch post states the caveat plainly, in a sentence that rarely survives into the summaries:

When the response does not include a refusal and the model’s response has not been prematurely interrupted (as indicated by finish_reason), then the model’s response will reliably produce valid JSON matching the supplied schema.

The headline deserves the same care. OpenAI’s launch post reports that gpt-4o-2024-08-06 scores “a perfect 100%” on their complex-schema-following eval, versus under 40% for gpt-4-0613. The sentence before it explains how:

despite this model’s performance improvements (93% on our benchmark), it still did not meet the reliability that developers need… So we also took a deterministic, engineering-based approach to constrain the model’s outputs to achieve 100% reliability.

The model reached 93% and the grammar mask closed the remaining seven points. That’s a real engineering achievement, and it also says the unconstrained model produced non-conforming output on about 7% of that eval. The constraint holds the model in place rather than teaching it, and since the eval isn’t public, the 100% is a vendor claim on an unnamed corpus.

vLLM’s tool-calling docs make the same point about a different failure axis, more bluntly:

You are guaranteed a validly-parsable function call - not a high-quality one.

Workload. Four schema families mirror what agent pipelines emit, at mixed output lengths:

• a flat classification result
• a nested tool call
• an enum-heavy triage decision
• an array-of-objects extraction

The same four families feed both the truncation-recovery measurement and the backend-conformance probe below, and the concurrency ramp cycles three prompt templates against this schema set. Single-shot question answering under-represents production agent traffic, so three further task sets run against the same ramp. An edge set carries keywords at xgrammar’s enforcement boundary. An agent set adds a genuinely two-turn task: a schema-constrained tool call, a synthetic tool result spliced into the transcript, then a schema-constrained answer turn. A five-step agent task adds two tool calls, an arithmetic step over both results, a planning step, and a final answer that has to carry the earlier turns’ values through.

Arms. The study runs five arms, of which four were measured:

1. vLLM on an H100 GPU Droplet with structured outputs in strict mode, backend explicitly pinned rather than left on auto.
2. The same schemas and prompts with the schema stated only in the prompt text and no grammar enforcement, which is the baseline everyone starts with.
3. DigitalOcean Serverless Inference with native structured output, measured against mistral-3-14B in strict mode. This is a managed, black-box endpoint where the backend, engine version and batching policy are provider-internal, and it runs specifically to contrast a stack you can inspect against one you can’t. Its prompt_only baseline wasn’t run, and because the model differs from arm 1’s Qwen2.5-7B it’s a second data point on enforcement consistency rather than a controlled isolation of the serving layer.
4. Arm 1 again with VLLM_XGRAMMAR_CACHE_MB reduced from its 512 MiB default to 1 MiB, to find out what an undersized grammar-compiler cache costs. The server’s environment was verified per restart, and a return-to-default control run confirms the restart itself changed nothing.
5. A second self-hosted engine, SGLang, which remains optional and unrun.

The ramp. Concurrency runs 1 → 10 → 50 → 100 as fixed-count bursts, then 1 → 400 as sustained 180-second windows. The reason for the second ladder is that 100 sits below the server’s 128-slot batch capacity, and a ramp that never queues can’t detect queueing.

Each level records the following:

• schema validity rate, meaning the response parses and validates against the schema
• semantic validity rate, from field-level checks against ground truth where the task allows
• truncation rate, read from termination metadata
• one normalized outcome per request, with a taxonomy category for structured-output failures
• end-to-end p50 and p99
• client-side TTFT, on a separate streaming pass
• measured cost per usable output, from real token counts
• per-token decode time, derived from the three preceding measures
• server-side contention signals scraped across the measured window, namely queue depth, KV-cache occupancy, and preemption count

A second axis ramps distinct grammars rather than requests, from 64 to 2,048 schemas, against both the default compiler cache and an undersized one. Request count can’t reach compilation; cardinality can. The instrumentation gap left is profiling inside the sampling loop, so the ramp can measure what enforcement costs per token end to end but can’t separate mask computation from scheduling within that. Why your vLLM p99 latency blows up in production covers the contention mechanisms, chunked prefill, scheduling and preemption, whose validity-side effects this section looks for. Metrics that Matter with Serverless Inference is the measurement-protocol standard (trial counts, warm-up discards) this ramp follows, and the Ornith 9B benchmark is the source of the async concurrency harness pattern it reuses.

Methodology discipline. The vLLM structured-output API is actively churning, in three ways that break copied advice:

• guided_json, guided_regex, and guided_decoding_backend weren’t deprecated, they were removed in v0.12.0 and replaced by structured_outputs.
• Strict tool calling landed in June 2026.
• reasoning_content was renamed reasoning, so client code can silently read an empty field.

A large share of the structured-output advice in circulation describes an engine surface that no longer exists. So every result below is stamped with the exact engine version, the resolved grammar backend and its library version rather than just the requested one, the tokenizer, and the sampling parameters. “Validity was X at vLLM 0.27.1 with xgrammar” doesn’t identify a build, because 0.27.1 accepts a range of xgrammar versions. “With xgrammar 0.2.3” survives an upgrade. Everything here is true of one specific release and some of it will be wrong within two, so treat the numbers as stamped rather than standing.

If you’re on the DigitalOcean vLLM 1-Click Model, it deploys vLLM on a GPU Droplet with no additional setup, but you still need to pin the structured-output backend yourself once it’s up. The marketplace image doesn’t do that for you.

Pre-committed to the null. If strict-mode validity holds under load and only latency degrades, that gets reported as the finding rather than as a disappointing result to explain away. “The failure folklore about concurrency is misattributed, and here’s what actually degrades” is exactly as citable as a validity collapse would have been, and it’s what the measurement below shows. Holding to that commitment also meant reporting, once the instrumentation existed, that the first version of this ramp had never queued a single request and so had ceded the question rather than answered it.

Three limits on the design, collected in one place. Each of these comes back where it bites, and none of them is discovered late:

1. Arm 5, SGLang, is sourced from code-reading rather than measurement, so every cross-engine statement below covers xgrammar against guidance on one engine.
2. Preemption never fired at any level in any arm, so the one contention mechanism with a plausible route into constrained decode is untested rather than cleared.
3. The tasks run at temperature 0 with a fixed seed except where stated, so several rates below are deterministic paths repeated rather than independent draws. Each one is flagged where it matters, and the varying-seed control under Results addresses the sample-size problem directly.

On retention: per-request outputs weren’t retained for the earliest vLLM ramp, though every run after it retained them.

The flat line at 0.67 is what no enforcement cost this task set, at every level equally. It’s a property of the task rather than of unenforced generation, since the five-step workload further down runs the same prompt-only arm at 1.00. The rest of this section covers how each line got there, plus four findings the chart can’t show, all of which sit on the flat part of it:

• what a repair layer does to a truncated document
• which schema keywords your backend only pretends to enforce
• the failure that enforcement moves rather than removes
• the cache misconfiguration that costs 2.5 times the latency without touching validity at all

Truncation, and what repair actually recovers

Truncation is measured two ways here: synthetically across every cut point, which is this subsection, and observed directly at a tight token budget, which is the H2 result further down.

One thing needs no experiment. Every document here is a top-level JSON object, so every proper prefix is missing its closing brace and can’t parse. That’s true by construction.

What a repair heuristic does to a cut-off document is measurable, and reaching for one is what every team does after their first truncation incident.

Using the four schema families from the experiment design, the experiment generates 200 documents for each, cuts them at approximate token boundaries, and runs each cut through repair. It then asks four separate questions, because collapsing them hides the result:

1. Does the repaired document parse?
2. Does it validate against the declared schema?
3. Does it pass field-level business rules?
4. Is it byte-identical to the document being written?

The repair does more than close brackets. It closes unterminated strings, drops dangling commas and colons, walks back over partial keys, then closes open brackets. It’s a custom heuristic, published in full in the accompanying script, because these numbers mean nothing without the exact implementation.

Two limits bound how far you can read this:

• Cuts land at regex-approximated token boundaries rather than the target model’s BPE vocabulary, so the axis is coarser than a real generation.
• Cut points are pooled per document, so these aren’t production failure probabilities. They describe what happens given a cut at a point of completion. Read them as a shape rather than a rate you’ll see in your logs.

The flat schema, by decile of document completion:

In the 60–90% band every repaired document parses, validates against the schema, and passes the business rules, and not one is the document the model was writing. A truncated summary is still a non-empty string of the right type, so schema validation has no opinion, and neither does a “field must not be empty” rule. The enum-heavy schema behaves the same way, and the nested one nearly so.

Pooled across all cut points, the share of schema-valid repairs that are not byte-identical to the original runs 96.2% (flat), 97.5% (nested), 96.9% (enum-heavy), and 50.0% (array-of-objects).

Array-of-objects is the instructive exception. Schema validity stays at 0.00 through nine deciles and reaches 0.07 in the last, because minItems and the required per-record fields reject a half-written record immediately. The schema that fails loudest protects you best.

Every automated check says yes from about 60% completion on, and the document still isn’t the one the model was writing. The right panel is the same measurement against a schema built to refuse.

So repair recovers well-formedness far more readily than content. If your termination metadata says the response was cut off, discard it rather than repairing it, and keep repaired documents away from validators that will wave them through.

Reasoning models make truncation worse. Both major vendors document the mechanism separately and neither connects the two halves. OpenAI, on reasoning tokens:

If the generated tokens reach the context window limit or the max_output_tokens value you’ve set, you’ll receive a response with a status of incomplete… This might occur before any visible output tokens are produced, meaning you could incur costs for input and reasoning tokens without receiving a visible response.

Anthropic’s docs make the same point about thinking tokens counting against the output budget, with the response ending at the max_tokens stop reason and a truncated or missing text block. This guidance is version-sensitive: current Claude models use an effort setting rather than a manual thinking budget, and budget_tokens is deprecated on 4.6 and rejected on 4.7 and later. Check which regime your model is in before copying a budget number from anywhere, including here.

Add a schema constraint and you get a request that spends its entire budget reasoning, emits zero JSON, and returns null content. Your parser reports a JSON failure; the model never reached the JSON. OpenAI suggests reserving at least 25,000 tokens when starting out with reasoning models, which is experimentation guidance for that model class rather than a universal floor for structured output or tool calls. It gets misquoted as one.

Detect truncation from termination metadata, not from null content. Null content also accompanies tool calls and other typed outputs, so treating it as a truncation signal misclassifies healthy responses. The field differs by API:

Generic finish_reason advice doesn’t survive contact with the Responses API, which doesn’t have that field at the top level. Write the check per provider.

The constraint boundary, where “strict” quietly isn’t

Every backend supports a subset of JSON Schema. The subsets differ, they change between releases, and the circulated lore about them is out of date. Two corrections are worth making, since both appear in posts still being cited in 2026.

vLLM no longer rejects pattern, minimum, or minLength. The list below is what vllm/v1/structured_output/backend_xgrammar.py explicitly rejects at preflight (identical at tag v0.27.1 and at HEAD), and it’s narrower than folklore suggests:

A rejection list isn’t a support matrix, so I stopped reading source and measured it. The probe submits 18 schemas, each with a prompt demanding a value that violates the constraint, five trials apiece, and records whether the decoder actually prevented it. It now covers two backends: restart the server with a different --structured-outputs-config.backend and rerun the identical probe. outlines and lm-format-enforcer were attempted too and are excluded below; both errored on every request in this environment for reasons unconnected to any tested keyword, covered after the backend-election discussion. Results on vLLM 0.27.1, xgrammar 0.2.3 and guidance (built on llguidance 1.7.6), temperature 1.0:

† On both backends the minimum trials that completed never violated the constraint, but 60% of attempts truncated before finishing even at 4x budget, which leaves too few completed trials to call enforcement either way. That result is unmeasured rather than evidence of anything.

The two backends split as follows:

• Under xgrammar, 9 of 18 keywords were enforced, 5 were honestly rejected up front, 3 were accepted but not enforced, and 1 returned no clean verdict.
• Under guidance, 14 of 18 were enforced, 3 were rejected up front, none were accepted but unenforced, and 1 returned no clean verdict.

For typed keywords on both backends the rejection list is an honest interface. Everything accepted was enforced, and everything unsupported was refused with an error rather than silently ignored. There were no silent failures among typed keywords on either backend.

Silent under-enforcement, meaning the backend accepts a constraint and then ignores it, appears exactly three times in this grid, and all three are fragments carrying no sibling "type" on xgrammar. Guidance had none. Everything else is either enforced or refused outright on both backends, and the two mid-grid disagreements are multipleOf and patternProperties, rejected by one backend and enforced by the other.

OpenAI’s supported subset has also expanded. pattern, format, multipleOf, minimum/maximum, and minItems/maxItems are all documented as supported for base models now; in August 2024 they were listed as unsupported. allOf, not, if/then/else, dependentRequired, and dependentSchemas remain unsupported, but that failure is loud: “you will receive an error.”

The boundary problem lives in two quieter places, and the second turns out to belong to one backend rather than to constrained decoding generally.

The type gate, and it’s xgrammar’s specifically. Every branch of vLLM’s xgrammar preflight check (has_xgrammar_unsupported_json_features) tests obj.get("type") == ... before examining the keyword. A fragment like {"minimum": 100} with no sibling "type" bypasses that gate and reaches XGrammar’s compiler, where handling is also type-driven, so the constraint falls through both layers.

The probe confirms it cleanly, because each untyped case in the grid above has a typed twin in the same run. {"type": "string", "pattern": "^[A-Z]{3}-[0-9]{4}$"} held 5/5 on both backends; strip the sibling to {"pattern": "^[A-Z]{3}$"} and xgrammar violated it 5/5 while guidance still held 5/5. maxLength did the same thing, and so did an untyped {"minimum": 100}. One keyword and one request shape produced two outcomes on two backends. The gate belongs to xgrammar’s implementation rather than to vLLM generally or to constrained decoding as a concept, because guidance didn’t key enforcement off a "type" sibling the same way and none of its five untyped trials slipped through. Draft 2020-12 schemas omit type routinely, and so does anything generated by a loose conversion from TypeScript types or Python dataclasses. If you’re on xgrammar, audit your schemas for untyped constraint fragments and treat any you find as unenforced. If you’re on guidance, this failure mode wasn’t reproduced in five trials, which is evidence it’s less likely, not evidence it’s absent. Audit anyway.

Backend choice also changes what’s supported at all, independent of the type gate. multipleOf and patternProperties sit on xgrammar’s rejection list, so the server refuses the schema outright, which is the honest failure mode. Under guidance, both schemas were accepted and the constraint held 5/5. The vLLM version and the installed library versions were identical, only the backend differed, and one of them refuses a schema the other enforces cleanly. Pinning the backend isn’t only about the type gate, because “my schema works” and “my schema is rejected” can both be true of the identical schema, depending on which of the four available backends the auto latch happens to pick.

One aside from the same run, which connects back to token budgets and, unlike the type gate, generalizes across both backends rather than belonging to one. On the cases that conflicted hardest with the instruction (minimum: 100 against “set qty to 1” on both backends, plus minLength, minItems, and maxItems on guidance), enforcement held on every request that finished, and 20–60% of attempts never finished even at four times the original budget. The decoder can’t emit a closing brace until the constraint is satisfiable, so a model pushing against it generates until the budget runs out, regardless of which backend compiled the grammar. Enforcement working and the request being unusable coexist on both backends measured here, which is a second reason to alert on truncation rate separately from validity rate no matter which one you’re running.

The backend election. The default backend is auto, which resolves per request. It tries xgrammar, falls back to guidance on a ValueError, and falls back to outlines instead if the schema uses patternProperties or the tokenizer is non-tekken Mistral. That much is reasonable. But vllm/v1/structured_output/__init__.py carries this comment:

# Initialize the backend the first time it is needed.
#
# NOTE: We only support a single backend. We do NOT support different
# backends on a per-request basis in V1 (for now, anyway...).
# _backend is set in Processor._validate_structured_output

The engine reads the resolved backend only when self.backend is None. Every subsequent request still computes its own _backend, and that value is then discarded. The first structured-output request your server ever sees elects the grammar backend for the lifetime of the process. If that was a health check carrying a patternProperties schema, your entire fleet is running outlines. Nothing is logged, because the fallback is a try/except ValueError that swallows the exception. (That last comment line is itself stale, incidentally: Processor is now a shim for InputProcessor, and the resolution logic lives in vllm/sampling_params.py.)

Two consequences follow:

• Later requests may be enforced differently than you expect, and they may also fail to compile at all against the latched backend, since each backend’s coverage differs.
• Falling back to outlines isn’t the same as being enforced by outlines, because patternProperties may be ignored or fail compilation in current outlines-core. A fallback path is a change of failure mode rather than a guarantee of coverage.

The escape hatch is harder to reach than it used to be. The colon-suffix xgrammar:no-fallback form went away in April 2025 (PR #17008), replaced by a dedicated --structured-outputs-config.disable_fallback flag, which survived v0.12.0 and has since been removed on main. Pin --structured-outputs-config.backend explicitly instead.

A startup-order dependency is how the same schema enforces differently in staging and production with identical engine versions, identical flags and no error anywhere. The exposure isn’t hypothetical here either. The serving machine for this article’s ramp had all four backends installed and importable: xgrammar 0.2.3, outlines_core 0.2.14, llguidance 1.7.6 and lm-format-enforcer 0.11.3. Under auto, the first request through the door picks one of them, and whichever three you didn’t pin against are backends you probably didn’t test. Pinning the backend costs one flag.

Two of those four are worse than untested here, because on this exact install outlines and lm-format-enforcer return HTTP 500 on every request, immediately, for reasons that have nothing to do with the schema:

• The outlines backend rejects the model’s own end-of-turn token when the grammar tries to close the document.
• The lm-format-enforcer==0.11.3 vLLM integration imports a module path (vllm.transformers_utils.tokenizer.MistralTokenizer) that no longer exists after vLLM 0.27.1’s internal refactor.

If the first structured-output request your process handles happens to elect either one, every subsequent request fails with a generic 500 until you restart with the backend pinned. So auto here wasn’t choosing between four tested options. This run can’t establish whether that composition, meaning which backends are installed and which of them are broken, generalizes to your install. Check your own pip freeze and probe your own backends before assuming any of it transfers.

The measured version of this problem is in JSONSchemaBench (Geng et al., arXiv:2501.10868, 2025, a non-archival ES-FoMo workshop paper), which ran 9,558 real-world schemas through every major framework and separated declared coverage from empirical coverage. Two things bound the table:

• Their figures come from XGrammar 0.1.6 and Outlines 0.1.8, where this article’s host resolved xgrammar 0.2.3 and outlines_core 0.2.14, so read it as a snapshot of a moving target.
• The rows use different underlying models, so it isn’t a clean head-to-head.

On their GitHub-Hard split:

Their reading of the OpenAI row is the useful one:

While closed-source implementations have low empirical coverage, they have very high compliance rates, indicating that their providers have taken a more conservative strategy, implementing only a subset of JSON Schema features that they can reliably support.

On this split OpenAI declares the least and delivers on what it declares. The prompted-LM row shows 1.00 declared coverage for a mechanical reason: with no compiler in the path nothing rejects a schema, so “declared” is definitionally 1.00 rather than a claim anyone made. Its 0.13 empirical figure is what argues against prompt-only JSON.

The bench also probes the cost Willard & Louf assumed away: on Outlines, “JSON Schema features like minItems, maxItems, enum, and Array, while supported, often take 40 seconds to 10 minutes” to process. That is their word, covering compilation and generation together under a timeout, not compilation alone.

Semantic garbage inside valid syntax

Decoding constraints can’t reach factual or cross-field correctness. Asked to itemize a four-line invoice under a strict schema, Qwen2.5-7B returned well-formed records and a total of 38.58 against line items summing to 34.65, and it did that on every request that drew the template (the raw output is under Specimens, below). No decoding constraint can reach it, because 38.58 is a perfectly good number of the right type in the right field. OpenAI concedes the point in the same launch post that announced the guarantee: “the model may still make mistakes within the values of the JSON object.” An arithmetic check catches it immediately.

The published version of the same result is the Structured Output Benchmark (Singh, Khurdula, Khemlani, Agarwal, arXiv:2604.25359, April 2026, preprint, treat accordingly): models achieve “near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio.” The numbers need two clarifications. The image and audio figures come from OCR-derived text and transcripts rather than raw media, and since compliance is near-perfect rather than perfect, an aggregate value-accuracy figure can’t establish that every wrong value sat inside a schema-valid document. The direction is clear, but the universal version isn’t supported.

My truncation results are a special case, where repair produces documents that pass every automated check without reconstructing the content. The general case doesn’t need truncation at all. Enum fields are the one place the mask does semantic work: an out-of-set value is unsamplable, which makes enums the most reliable thing you can put in a schema. They still don’t stop the model from confidently picking the wrong valid member, which is a different and unaddressed problem. A free-form string field constrains nothing beyond the quotes.

Whether the constraint itself hurts reasoning is contested, and the two most-cited papers point in opposite directions.

Tam et al. (EMNLP 2024 Industry Track, arXiv:2408.02442) reported dramatic collapses under format restriction: GSM8K on GPT-3.5 going 76.6 → 29.87, Last Letter on Gemini-1.5-Flash going 65.4 → 0.67. Two things get dropped when those numbers are quoted:

• The mechanism is ordering rather than constraint. In their words, “100% of GPT 3.5 Turbo JSON-mode responses placed the ‘answer’ key before the ‘reason’ key, resulting in zero-shot direct answering instead of zero-shot chain-of-thought.”
• The collapsing arm is provider JSON mode rather than schema-driven grammar decoding.

The paper’s own grammar-constrained arm is far milder. gpt-4o-mini scores 94.57 on GSM8K in natural language against 91.71 with JSON Schema, and on Last Letter the schema arm wins, 86.07 to 83.11.

JSONSchemaBench found the opposite sign, reporting that “constrained decoding, regardless of the framework, achieves higher performance than the unconstrained setting,” with GSM8K going 80.1% (LM-only) to 83.7% (XGrammar). “The Format Tax” (Lee, D’Antoni, Berg-Kirkpatrick, arXiv:2604.03616, April 2026, preprint) splits the difference: “format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied,” with grammar constraints lifting compliance from 55.7% to 92.2% while accuracy stays flat at 57.3% → 55.7% against 61.5% freeform. That aggregate covers single-turn, thinking-off runs on selected open-weight models, and newer closed models sometimes show no tax at all.

So asking for JSON costs some reasoning while enforcing the JSON you asked for costs little more. If you’re losing accuracy the prompt is the likelier place to look than the decoder, and the plausible fix is to let the model reason before it formats, either in a thinking span the grammar doesn’t cover or in a reasoning field ordered before the answer fields. That last move gets stated as settled and isn’t, so it carries three caveats:

• Key-order preservation is an OpenAI guarantee (“outputs will be produced in the same order as the ordering of keys in the schema”) rather than a cross-provider one, so verify it on your stack.
• Tam et al. observed a correlation between key ordering and the accuracy drop without running a controlled reordering intervention, so “put reasoning first and you recover most of the loss” is a hypothesis rather than a result.
• A visible reasoning field isn’t equivalent to hidden reasoning tokens. The mechanism differs and so does the cost.

Contention, and what the concurrency ramp actually showed

Schema validity was measured against concurrency twice: first as a bounded burst to concurrency 100, then as sustained load to concurrency 400.

Why the first ramp couldn’t answer the question it was built for. The server ran max_num_seqs=128 and the ramp stopped at concurrency 100, so every request it ever offered fit inside a single batch. Nothing queued, nothing was preempted, and instrumentation added later measured KV-cache occupancy at 0.05% with 4 requests in flight and 0.97% at concurrency 100. “Concurrency did not change validity” was a true statement about a server that was never contended. Scheduler queueing wasn’t absent by observation, it was impossible by construction, and no number of extra requests at concurrency 100 would have changed that. The fix is a ramp that crosses the batch capacity.

The design limits bound the first table below. Its task set is three prompt templates at temperature 0 with a fixed seed, so 200 requests per level are 200 repetitions of three deterministic completions. There are three effective samples, so nothing there estimates a population failure rate, and “0.67” means “two of three templates always succeed” rather than a proportion with a confidence interval. Two hundred requests at concurrency 100 is also about two request waves. The sustained tables fix the burst problem with 180-second windows per level, giving 8,124 requests at concurrency 400 against the burst ramp’s 200, but those extra requests buy duration and queueing coverage rather than independent samples, because they are the same three completions repeated. The varying-seed run further down is what addresses the sample-size problem.

Setup. The self-hosted arm ran on the following configuration:

• vLLM 0.27.1 with Qwen2.5-7B-Instruct, on a single H100 80GB
• xgrammar 0.2.3, pinned explicitly rather than left on auto
• max_num_seqs=128 and max_completion_tokens=512
• temperature 0.0 with a fixed seed, 200 requests per level, warm-up discarded

vLLM 0.27.1 specifies a range for xgrammar rather than an exact pin, so the library version is part of the identifier rather than a footnote. Per-request outputs and server logs are among the things not retained, so these tables are a record of one configuration rather than a replayable artifact.

Strict mode, the original bounded burst, 200 requests per level:

Prompt-only baseline, same tasks, same schemas, no enforcement:

Now the same arm under sustained load, 180-second windows per level with 30 seconds discarded as warm-up, ramped past the 128-slot batch capacity. The three rightmost columns are the ones the first ramp never had, and they’re what make the validity column interpretable:

Every validity miss in that table is a dropped HTTP connection rather than a schema failure. The failure categories account for the deficit exactly: 4 failures against 6,837 requests at concurrency 100, 13 against 7,939 at 200, and 7 against 8,124 at 400. Every one lands in the harness’s transport bucket as RemoteProtocolError: Server disconnected without sending a response. Truncation is 0.00 at every level, and there isn’t a single constraint_boundary or extraction_parser event anywhere in the ramp.

So the claim strengthens, with its denominator stated. Among requests that returned a response at all, strict-mode schema validity was 1.00 at every level from concurrency 1 to 400, including at 400 with 270 requests queued behind the batch and p99 grown from 2.2 to 11.3 seconds. The sub-1.00 entries fold the dropped connections into the denominator, and the control run below removes them. Queueing was real this time, and it moved latency by roughly fivefold without moving validity at all. That’s consistent with the mechanism: the mask applies per token regardless of batch size, so waiting in a queue has no route to make a token illegal.

The two panels are the same requests measured two ways. The hollow marker is the pooling-disabled control, sitting on the same axis as the pooled p99 it replaces. Removing connection reuse lowered p99 by 0.718 seconds and narrowed the p99-minus-p50 spread by 39%.

What still hasn’t been tested, and it isn’t for lack of trying. Preemption never fired, staying at zero at every level including 400, and KV-cache occupancy peaked at 1.51%. A follow-up run at max_completion_tokens=2048 was built specifically to force cache pressure and failed to do so. Mean completion length came back at 144.7 tokens at concurrency 100 and 149.0 at 400, because these prompts don’t produce long documents and the budget was never the binding constraint. Cache occupancy peaked at 1.58%, preemption stayed at zero, and validity came in at 0.9999 and 0.9864, again entirely from transport drops. So the preemption-and-recompute interaction with grammar state, the one contention mechanism with a plausible route to corrupting constrained decode, remains unmeasured. Reaching it needs prompts that genuinely emit thousands of tokens, or a model large enough that the cache binds, rather than a larger budget.

The transport failures deserve their own note, because of who gets blamed for them, and because they turned out to be mine. They appeared only after concurrency reached 100, at 0.06%, 0.16% and 0.09% across concurrency 100, 200 and 400. A team folding those drops into an “invalid output rate” dashboard could still reach for the schema, the backend, or the model. None of those is involved: re-running concurrency 400 with connection reuse disabled and nothing else changed removed the failures:

The queue is still there, with 266 requests waiting against 270 in the pooled run, so this is the same contention with the client’s pool taken out of the path, and the failures vanish completely. Strict-mode schema validity at concurrency 400 under sustained load, with a real queue, is 1.00 with no asterisk. At a p50 near ten seconds, pooled connections could sit idle longer than the server’s keepalive timeout, so the client could check out a connection the server had already closed and get RemoteProtocolError for its trouble.

The control also lowered p99 from 11.279 to 10.561 seconds and completed 57 more requests, small differences compared with the earlier estimate but in the same direction. This is the misattribution the taxonomy exists to prevent, occurring inside this article’s own measurements: a transport failure that sat in the same dashboard cell as schema violations and had nothing to do with structured output at all. Before concluding that load broke your constrained decoding, check your client. Two bounds hold on that attribution:

• The control ran at concurrency 400 only, so the smaller deficits at 100 and 200 are pinned on the pool by their identical error signature rather than by re-running those levels.
• The race is timing-sensitive rather than a guaranteed consequence of a deep queue. A later sustained run at concurrency 400 with pooling still enabled, on the cardinality task set below, produced zero transport failures with 268 requests waiting, while the five-step agent arm lost conversations to the same drops at concurrency 100 with nothing queued at all.

Queue depth is neither necessary nor sufficient, which is why this rate belongs on its own dashboard row rather than read as a load signal.

A second signal, from removing the determinism. Rerunning the strict arm with a varying seed instead of a fixed one gives 200 independent draws per level at temperature 0.7, so the rates become proportions rather than three repeated templates. Schema validity came in at 1.00, 0.995, 1.00 and 1.00 across concurrency 1, 10, 50 and 100; the single miss was a parse failure at concurrency 10. Semantic validity stayed between 0.665 and 0.670. So the semantic failure rate isn’t an artifact of three deterministic samples, because under genuine independent draws it lands in the same place. That makes the invoice-arithmetic failure a real property of the task mix rather than a repeated coincidence.

The same request, on a managed endpoint

The vLLM null is a claim about one pinned, inspectable stack. Running the identical harness, with the same schemas, the same strict json_schema requests and the same ramp, against DigitalOcean Serverless Inference (mistral-3-14B) answers a different question. Does a managed endpoint, where the engine version, backend, batching policy and max_num_seqs are all provider-internal, behave the same way? It does not.

The table below is strict mode on DO Serverless at max_completion_tokens=1024, raised from 512 after a first run showed mistral-3-14B’s natural output length hitting the smaller budget. The two runs are independent repeats at 1024.

Both runs tell the same story in the same direction. Validity is perfect and truncation is zero at concurrency 1, sliding to about 0.85 validity and about 0.15 truncation as load rises. The per-level numbers wobble between runs and the curve isn’t strictly monotonic, since run 1 reads 0.835 at concurrency 50 and 0.86 at 100. That is expected on a shared, autoscaling endpoint whose latency swung independently of offered load, with p99 ranging 6–15 s and no clean relationship to concurrency. What reproduced across two independent runs is the degradation from the concurrency-1 baseline, and that rules out one-off cluster noise. Every failure was a finish_reason: length truncation rather than a schema or transport error, so the rate limiter wasn’t contaminating the concurrency axis.

You can’t observe the mechanism from the caller’s seat. Mean completion tokens fell under load, from 500 at concurrency 1 to about 396 at concurrency 100, despite a 1024 budget the concurrency-1 case proved was ample. So something on DO’s side is cutting generations short under concurrency, and it could be a server-side generation cap, a dynamic per-request token limit, or load-sensitive backend routing. Those are indistinguishable from outside. What the two arms establish together is that a managed endpoint answers the load question oppositely to pinned self-hosting, because the load-induced truncation that vLLM didn’t exhibit appears here on the same request, and the managed layer gives you no way to attribute it. The model differs too, mistral-3-14B against Qwen2.5-7B, so these are two data points on enforcement consistency rather than a controlled isolation of the serving layer.

On latency, two signals and one remaining gap. Strict-mode p99 rose 59.5% across the original burst ramp against the baseline’s 26.7%, and under sustained load to concurrency 400 p99 went from 2.232 to 11.279 seconds while validity held. Neither of those separates grammar overhead from ordinary queueing.

A streaming pass does get closer, because time-to-first-token is where grammar compilation and mask setup would land if they land anywhere. Client-side TTFT p50 in strict mode runs 0.433 s at concurrency 1, 0.440 at 10, 0.480 at 50 and 0.656 at 100, a 52% climb. The prompt-only baseline over the same levels reads 0.433, 0.438, 0.489 and 0.611. At concurrency 1 the two arms are 0.3 ms apart, which is the cleanest statement available here that constrained decoding is close to free when nothing is queued, and it matches XGrammar’s own published TPOT figures in direction if not in magnitude.

TTFT still can’t attribute the divergence under load, because the arms emit different text and there’s no per-task stratification. Per-token decode time gets closer and is a steadier instrument. Subtracting TTFT from end-to-end latency and dividing by completion tokens gives the average cost of every token after the first, on the same prompts and the same offered load, differing only in whether decoding is constrained. The table runs six repeats per cell, each a median of roughly 200 requests.

Enforcement is close to free on an idle server and costs about a third of your per-token time once the batch is wide, and the cost grows with batch width rather than sitting flat. That is the shape you’d expect if mask application is per-request-per-step work competing across a widening batch.

Read it as an end-to-end operator cost rather than kernel time. Per-token decode under batching includes waiting between tokens, so part of the gap is strict requests occupying the batch longer and contending more, which is a real cost to whoever pays the bill but isn’t mask arithmetic in isolation. The concurrency-1 row is a single run rather than six, and the arms produce different text, so token counts differ. Separating mask computation from scheduling still needs a profiler inside the sampling loop. Why your vLLM p99 latency blows up in production documents chunked prefill, scheduling and preemption as the usual tail-growth mechanisms on this engine, and grammar overhead looks like an addition to that list rather than a rider on it.

Two hypotheses the first ramp couldn’t test. The 512-token, well-supported-keyword configuration above induced neither budget pressure nor constraint-boundary failures, so two of the folklore’s load hypotheses had nothing to bite on. Two runs built to provoke them, one at a tight max_completion_tokens=128 and one on an edge-schema set carrying keywords at xgrammar’s enforcement boundary, resolve both in opposite directions.

Truncation doesn’t rise with concurrency at a fixed budget (H2, refuted). At max_tokens=128, truncation held at exactly 0.665 at every level from concurrency 1 to 100, and a re-run on the instrumented harness reproduced it to the request, giving 133 truncations and 67 clean passes out of 200, identical at concurrency 1, 10, 50 and 100. The rate is fixed by the task rather than the load, because at temperature 0 the two longer templates exceed 128 tokens deterministically and the short one fits, so two of three truncate regardless of batch size. A tightened budget raises the truncation floor, and concurrency doesn’t move it. One caveat applies: determinism structurally forbids a load effect here even if a small one existed, so detecting a subtle preemption contribution would need varied prompts. But the strong claim, that load drives truncation up, doesn’t hold on pinned self-hosted vLLM, and the sustained ramp adds to that with truncation at 0.00 out to concurrency 400 on the default budget.

Constraint-boundary failures are a load-independent constant offset (H3, confirmed). An edge task set of three schemas, comprising an untyped pattern fragment, a multipleOf integer, and a typed-string control, produced an identical failure split at every concurrency level:

• 67 silent under-enforcements, where the untyped ^[A-Z]+$ pattern wasn’t enforced, the model returned lowercase, and only the downstream validator caught it
• 67 request-time rejections, where xgrammar refused the multipleOf schema with an HTTP 500
• 66 clean passes, from the typed control, where the sibling "type":"string" let the gate fire

That split was identical to the request at concurrency 1, 10, 50 and 100 rather than approximately constant, and it reproduced on the instrumented harness. The contrast between two requests differing only by a "type" sibling is the type-gate finding from What the guarantee covers holding across the full ramp. As with the truncation hypothesis, the outcome is schema-determined and therefore structurally constant, so what the run establishes is that load adds no variance to it.

Enforcement relocates a failure rather than removing it. Both arms failing at 33% on the default set proves nothing by itself, since one-of-three failing in each arm is arithmetic coincidence rather than a conserved quantity. The stronger claim needs a task set built for it, using the same prompts across both arms and checking whether the specific items failing in one arm pass in the other. The table below covers a four-task set at concurrency 100, with 50 requests per task per arm.

Three tasks pass in both arms, and the fourth fails in both arms at a different rung each time. Under no enforcement the invoice task wraps its answer in a markdown fence and json.loads dies on the leading backtick, so schema validation never runs. Under strict enforcement the fence is unsamplable, the document parses and validates, and the same task fails at business rules on the arithmetic instead. The prompt, the schema and the model are identical across both. The failure didn’t disappear, it moved from the rung that stops a pipeline loudly to the rung most teams never build.

Two limits govern how far to read that:

• Temperature is 0 with a fixed seed, so this is one deterministic item observed 200 times per arm rather than a rate over independent draws. It is a clean existence proof that enforcement can relocate a failure rather than an estimate of how often it does.
• Relocation is task-specific. The five-step agent task described below passed cleanly in both arms, so enforcement changed nothing there, and adding it to the four-task set as a fifth item reproduced the same split at every level, with array_extract still the only task failing.

One task relocates and four don’t, which is the useful shape of the result rather than a weakening of it. If your enforcement rollout made your parse-error dashboard go quiet, that isn’t by itself evidence that anything got better.

One denominator is worth keeping straight in the three-template runs. Cycling three templates across 200 requests gives 67 / 67 / 66, so the invoice task ran exactly 66 times per level there, and 50 times per level in the four-task set above.

Prefill interference is the usual explanation for tail growth under load. Sarathi-Serve measured a prefill iteration on Falcon-180B taking ~1150 ms against a ~200 ms decode iteration (Agrawal et al., OSDI 2024), which establishes the mechanism without explaining this ramp, since it used a different model on different hardware with no pipeline parallelism.

Adjacent DO work points at a failure the ramp was too short to catch. In the Ornith 9B serving benchmark on an H200, throughput scaled from 187.63 to 3,181.02 tok/s from concurrency 1 to 20, and one of 100 requests at concurrency 20 returned zero tokens. Alert on that separately, because retained status, finish metadata and usage counts distinguish it from malformed JSON, provided you log them.

Schema cardinality is the one contention mechanism this ramp could not reach, because every level offered the same three or four schemas.

Schema cardinality, and the cache that does bite

Grammar compilation is real work with a real cache. vLLM constructs xgr.GrammarCompiler(tokenizer_info, max_threads=8, cache_enabled=True, cache_limit_bytes=VLLM_XGRAMMAR_CACHE_MB * 1024 * 1024), and the comment in vllm/envs.py sizes the default: “The default of 512 MB should be enough for roughly 1000 JSON schemas.” The comment says MB while the code multiplies by 1024², so the practical limit is 512 MiB, about 524 KiB per schema in an LRU-bounded cache.

Every ramp above offered three or four schemas. They compiled once during warm-up and every measured request was a cache hit, which is why compilation never appeared in any latency number. Ramping request count can’t reach that mechanism. Ramping the number of distinct grammars in flight can.

A generator synthesizes N structurally distinct schemas of 3 to 7 fields each, varying field names, types and enum sets, with every string bounded by maxLength and a prompt that supplies an explicit value for every field so the model copies rather than invents. Requests round-robin the set at three per schema, so the second and third reads land as cache hits against the first read’s compile. The harness reports how many distinct schemas a level actually reached, because a run whose rotation stops short tested a lower cardinality than its label claims. The table below runs at concurrency 50, streaming, on the default cache.

A 32-fold increase in distinct grammars moves TTFT p50 by 9 ms and p99 by nothing that reads as a trend. Holding cardinality at 2,048 and sustaining load instead gives the same answer with a queue attached, at 5,066 requests at concurrency 100 and 5,855 at 400, schema and semantic validity of 1.00 at both, 268 requests queued at the top, and preemption still at zero. Cache churn and a deep queue arrived together there, which is the one configuration in this study that could plausibly have forced preemption, and it still didn’t fire.

Compiling a fresh grammar costs less than the measurement can see. Within a single run, a request whose schema has never been served before can be compared against requests for a schema already in cache, which holds prompts, load and server state constant and varies only the cache hit. Excluding each run’s first concurrent wave, which is inflated for an unrelated reason covered below, the cold-minus-warm difference in TTFT p50 across six runs comes to +2.3, +3.2, +0.1, −1.2, +2.0 and −6.9 ms. The mean is about zero and the sign flips, which bounds a fresh compile of one of these schemas below roughly 3 ms. XGrammar-2 (Li, Dong, Wang, Xu, Jiang, Chen, Proc. ACM Conference on AI and Agentic Systems, 2026) reports “about 10 ms, while XGrammar needs more than 1000 ms to compile” on their CONFETTI workload. Both can be true, because compile cost scales with schema complexity and a 3-to-7-field flat object is nowhere near a dynamic agentic grammar. A bound measured on small schemas licenses nothing about deeply nested ones.

Undersize the cache, though, and the same workload falls apart. The table below restarts the server with VLLM_XGRAMMAR_CACHE_MB=1, verified in the process environment rather than inferred from a flag, and changes nothing else.

That is two and a half times the TTFT and twice the per-token decode cost, and validity doesn’t move. Under sustained load it’s worse. At concurrency 400 the server completed 1,609 requests in the same 180-second window that produced 5,855 at the default cache, a 3.6-fold throughput collapse, with p50 going from 12.7 to 50.5 seconds and p99 from 15.9 to 68.8. Truncation stayed at zero, preemption stayed at zero, and every output was schema-valid and semantically correct.

So a cache misconfiguration costs 2.5× the latency and two thirds of the throughput while every rung of the validation ladder reports success. The symptom that does show up is slow generation, which is the one people attribute to the model or the GPU.

The penalty is flat across the top of the ladder, at 1.238, 1.231 and 1.229 seconds of TTFT for 256, 1,024 and 2,048 schemas. Once the cache stops fitting the working set, adding more schemas changes nothing, because every request is already paying. At the 2,048-schema rung, per-token decode rises from 24.43 ms with the default cache to 50.75 ms at 1 MiB. A separate three-repeat check at 512 schemas shows the same split, with default-cache medians of 24.15, 24.30 and 24.53 ms against 50.69, 50.92 and 51.02 at 1 MiB. Recompilation at under 3 ms can’t produce 740 ms of extra TTFT, so something the cache holds is being rebuilt and then consulted per token.

One label needs correcting, and it’s the sustained arm rather than the ladder. Its rotation reaches a fresh schema per request, so 1,609 completions touched 1,609 of the 2,048 schemas offered, which means that arm ran at a lower effective cardinality than its name claims. The throughput collapse is what the row measures and it stands. The flat-penalty reading rests on the fixed-count ladder rungs, which each reached their full offered cardinality, so nothing above depends on the sustained arm’s label being right.

The ratio panel is measured at 2,048 schemas, where the cache setting moves every latency and throughput figure and leaves validity at 1.00.

The threshold is bytes against working set rather than a schema count. The default taskset’s three schemas ran at 1 MiB with no penalty at all, matching the default cache to within the repeat spread, while sixteen synthetic schemas at the same 1 MiB paid the full 2.5×. The server and the setting were identical and only the working set differed. That also rules out a broken 1 MiB server, which would have been slow for three schemas too.

If the penalty is ordinary LRU eviction, those two observations bracket per-schema footprint between 64 KiB, because sixteen don’t fit in 1 MiB, and 256 KiB, because 2,048 do fit in 512 MiB. That would make the source comment’s implied 524 KiB conservative by a factor of two or more and put the default cache’s real capacity above 2,048. One measurement refuses that account. An earlier run of the same ladder at VLLM_XGRAMMAR_CACHE_MB=16 also showed no penalty at 2,048 schemas, which a 64 KiB footprint makes impossible. That run predates the provenance discipline the other two configurations were measured under, so it may be wrong, but it isn’t reconciled and the mechanism is therefore unresolved. The effect and its direction are measured on a there-and-back, since restarting at the default cache a third time returned TTFT to 0.484 s and per-token decode to 23.8 ms, so the degradation follows the setting rather than the restart.

So the practical guidance replaces a schema count with a measurement. A single agent with a dozen tools never comes close to any of this. A multi-tenant platform where each tenant registers their own tools can cross it, and the failure it crosses into is a latency and throughput failure that your validity dashboard will call healthy. The figure is also backend-specific, because 512 MiB describes XGrammar’s compiler cache and Outlines caches differently, so if your backend latched to something other than what you assumed, none of this describes your server. Measure TTFT and per-token decode time against a small working set and against your real one, and treat a gap between them as a cache-sizing problem rather than a model problem.

The first concurrent wave of a burst has inflated TTFT, and it has nothing to do with grammars. Offering 50 requests to an idle engine at once means they can’t all prefill simultaneously, so the back of the batch waits. In the unenforced arm that first-wave excess scales cleanly with burst width, running 0.2 ms at concurrency 1, 12.6 ms at 10 and 47.4 ms at 50, which is roughly a millisecond per request of burst. In the strict arm at concurrency 50 it runs 80 to 110 ms and, tested across three repeats at 16 schemas against three at 512, doesn’t depend on cardinality at all. A single 50-sample first-wave median varies by ±30 ms, which is enough to manufacture a cardinality trend out of nothing on short runs. If you reproduce any of this, discard the first wave or run long enough that it doesn’t dominate the median.

Multi-turn, which is what agent traffic actually looks like

Every ramp above is single-shot, and production agent traffic isn’t. A separate arm runs a genuinely two-turn task: a schema-constrained tool-call turn, a synthetic tool result spliced into the transcript, then a second schema-constrained turn for the final answer. The intermediate turn is validated on its own so a broken tool call is distinguishable from a broken answer.

It held completely, with schema validity of 1.00 and semantic validity of 1.00 at concurrency 1, 10, 50 and 100 across 200 requests per level, and an intermediate-turn failure rate of 0.000 at every level. The tool-call turn never malformed, and the answer turn carried the tool result’s provisioned_gb through to the response correctly every time. That’s the cross-turn version of the invoice check, and the one place in this study where a semantic rule the model could plausibly have failed came back clean.

Two turns is the shallow end, though, so a second arm runs five in one continuous transcript:

1. a usage tool call
2. a rate-card tool call
3. an arithmetic step over both results
4. a planning step
5. a final answer that has to carry values from turns one and two through to the end

Every turn is schema-constrained and validated on its own, turns three and four carry their own business rules, and the final check is a cross-turn one, because the answer must still hold the numbers the earlier turns established. That last rule is the one designed to fail if context handling degrades.

It held as well, and further than the two-turn arm did.

Six hundred conversations at concurrency 1 through 50 make three thousand model calls, and every one reached turn five with the cross-turn values intact. The single miss at concurrency 100 is a ReadError on the transport, carrying the same error signature as the client-pool failures elsewhere in this study, though this arm has no pooling-disabled control to prove it. It killed a conversation at turn two rather than corrupting one. There was not one content failure at any depth or any level.

The same task without enforcement is the more interesting half. Prompt-only also held schema validity at 1.00 through concurrency 50, across 3,000 model calls with the schema stated only in the prompt text. That locates the single-turn set’s 0.67 parse rate where it belongs, in one task wrapping its answer in a markdown fence rather than in a property of unenforced generation. Prompt-only isn’t reliably bad. It’s unpredictably fine, which is worse to plan around, because nothing about the two tasks told you in advance which one would fence its output.

What separated the arms was cost and tail rather than validity. Strict cost 25% less per usable output at every level from 1 to 50, a pure token difference since both arms sat at 1.00, which the cost section below works through. Latency inverted under load: prompt-only was 6% faster at concurrency 1 and 29% slower at 100, with p99 of 22.664 s against 15.401 s, and it lost nine conversations to transport drops where strict lost one.

Depth widens the tail, and that’s the clearest cost of multi-turn here. A conversation only finishes when its fifth turn does, so every turn’s delay lands in the same user-visible number. The p99-to-p50 ratio on the single-turn burst ramp was 1.09 at concurrency 1 and 1.33 at 100; on the five-step task it’s 1.27 and 1.84, and a five-step conversation’s p99 at concurrency 100 is 15.4 seconds against 3.6 for a single request at the same level. Read that as the end-to-end consequence rather than a decomposition, because the five turns of one conversation share a server, a batch and a growing transcript, so their latencies are correlated rather than independent draws. Transport behaves the same way, in that a conversation dies if any of its five calls dies, which points at why losses appear at concurrency 100 here against 400 on the single-turn arm. The counts don’t support a rate, though: one event in the strict arm and nine in prompt-only is a shape rather than a measurement.

Two limits stand:

• The transport drops arrived with vllm:num_requests_waiting at zero, so nothing queued at any level. Concurrency 100 running five-turn conversations still fits inside a 128-slot batch, which means this arm carries no contention signal at all.
• The failure the depth was built to catch, context growth pushing later turns against the token budget, didn’t appear, for a reason the data makes plain. Five turns of small outputs against a 512-token per-turn budget never approached it, with truncation at 0.00 throughout.

Reaching that second one needs turns that emit thousands of tokens, or a conversation long enough that the transcript itself binds, rather than five more turns of the same size.

Specimens, traced

This section shows every category from the failure taxonomy that occurred in these runs, along with the rung that caught it and the transport failure that sits outside the taxonomy. Truncation gets two specimens, because the second fails for a reason the first doesn’t cover. Elisions and abridgements are marked where they happen. Publishing the raw output rather than describing it matters because the interesting cases look completely healthy.

Semantic, inside valid syntax. This is array_extract in strict mode, returning finish_reason: stop and HTTP 200, with no error at any automated rung except the last.

{
  "records": [
    { "sku": "SKU-01044", "qty": 3, "unit_price": 10.00, "note": "block storage" },
    { "sku": "SKU-90210", "qty": 1, "unit_price": 4.41, "note": "H100 hour" },
    { "sku": "SKU-33127", "qty": 12, "unit_price": 0.01, "note": "bandwidth GB" },
    { "sku": "SKU-77219", "qty": 2, "unit_price": 0.06, "note": "snapshot" }
  ],
  "total": 38.58
}

There are four records, every field is present and correctly typed, and every quantity and price is copied correctly from the prompt. The line items sum to 34.65. The arithmetic rule caught this, and nothing else did.

Extraction and parser. This is array_extract on the prompt-only arm, complete and untruncated. The completion opens with a markdown fence instead of a brace, and the document inside is elided here but retained in the per-request dumps.

```json
{ ...well-formed JSON, elided... }
```

json.loads dies on the leading backtick, before it reads a byte of the document, so the parse rung caught this one. The model’s JSON is fine here, and a repair layer that strips fences would “fix” it, which is why the raw completion has to be logged alongside the parsed result to tell this apart from a parser bug.

Constraint boundary, silent under-enforcement. This is an untyped {"pattern": "^[A-Z]+$"} fragment against a prompt demanding lowercase, in strict mode, returning HTTP 200 and finish_reason: stop.

{"code": "abc"}

The backend accepted the schema and ignored the constraint, because xgrammar’s preflight gate tests for a sibling "type" before examining the keyword. Schema validation caught it downstream, after generation was already paid for, and the error was recorded as schema: pattern at ['code'].

Constraint boundary, honest rejection. This is a {"type": "integer", "multipleOf": 5} schema from the same run and the same ramp.

HTTP 500 — schema_rejected

There is no content at all, because xgrammar refuses the schema at request time. This is the good failure mode, sitting in the same category and the same run as the specimen above but at the opposite end of the honesty spectrum. The request itself caught it.

Truncation. This is nested_toolcall at a 128-token budget, returning finish_reason: length.

{"tool": "usage_query", "arguments": {"query": "block sto

It is grammatically valid as a prefix and unparseable as a document. Termination metadata caught it before the parser ever ran, which is why you check that field first.

Truncation, the variant enforcement creates. An unpublished pilot with the first version of the schema-cardinality generator produced schemas whose string and integer fields carried no bounds, against prompts that named the fields without supplying values. At a 512-token budget, 37% of those requests truncated in a way the ordinary budget story doesn’t cover. Its aggregate JSON was not retained, so that rate is not independently replayable from the repository. Below are two captured specimens from the pilot, both in strict mode, both returning HTTP 200 and finish_reason: length, and both abridged at the point the pattern is clear.

{"firewall_00006_f0": "firewall", "firewall_00006_f1": 1,
 "firewall_00006_f3": "nyc3_datacenter_id1234567890123456789012345678...

{"registry_00007_f0": "nyc3", "registry_00007_f3": "cus_18ab4f21",
 "registry_00007_f4": 20260700000000000000000000000000000000000000...

The second one is the interesting case. That runaway field is declared {"type": "integer"}, and the model emitted roughly 400 digits into it. Nothing here is a grammar violation, because one more character inside an open string is always a legal continuation, and so is one more digit inside a number. Greedy decoding at temperature 0, given a field name that carries no meaning and no value to copy, fell into a repetition loop, and the constraint had no way to break it because it was never being broken.

That’s the prefix guarantee arriving exactly as advertised. Extendable is not terminating: the mask can say which tokens are legal next but can’t say stop, and a field whose type admits arbitrary length has no point at which stopping is the only legal move. So an unbounded string or numeric field in an enforced schema is a truncation risk enforcement doesn’t remove, and may make easier to reach.

The fix is a bound plus a prompt that gives the field something to say. Rewriting the generator to put maxLength on every string and supply an explicit short value for every field produced 0.00 truncation across the 10,000-plus published requests of the cardinality ladder above, against 37% in the unpublished pilot. Choose the bound with the conformance grid in hand, though: typed maxLength and maximum are enforced on xgrammar, multipleOf is refused outright, and minimum returned no clean verdict, so the keyword you reach for determines whether you get a bound, an error, or nothing.

Termination metadata caught these too, same as the specimen above. Both validate as prefixes and neither is a schema violation, so a pipeline checking only “did it parse, did it validate” sees a parse error and reaches for a repair library, which is the worst available response.

Transport, which isn’t one of the five. There is no document at all, only a RemoteProtocolError: Server disconnected without sending a response at concurrency 400, and the HTTP client is what caught it. It appears here because it’s the failure most likely to be miscounted as a structured-output problem. It was the only rate in the sustained ramp that moved with load, which is what made it look like contention, and it also appeared at concurrency 100 in the five-step arm with nothing queued at all. The pooling-disabled control puts it in the client rather than the engine.

The cost dimension

Failed structured outputs are paid-for tokens plus retry tokens. The token bill is smaller than people assume. The latency cost is larger.

The model below rests on three inputs:

• DigitalOcean Serverless Inference pricing for Llama 3.3 Instruct-70B, at $0.65 per 1M input and $0.65 per 1M output tokens
• a representative agent turn of 2,000 input and 400 output tokens, which is $0.001560 per attempt, with up to three attempts
• the cost-per-token framework from Token Economics Across Traffic Profiles on Dedicated GPUs

A 4% failure rate costs you about 4% in tokens. If the token bill is your only concern, structured-output failure is close to a rounding error. Repair-prompt retries are the more expensive path in this model, because they resend the failed output plus the error as input on every attempt after the first, which prices them above full regeneration at every failure rate in the table.

The measured version, from actual token counts. The sustained ramp recorded prompt and completion tokens per request, so cost per usable output is arithmetic over measured tokens rather than an assumed failure rate. One pricing note applies. The self-hosted arm is billed per GPU-hour rather than per token, so these rows price its measured tokens at the same $0.65/$0.65 per 1M to make the arms comparable rather than to reproduce a bill.

Two things fall out that the modelled table couldn’t show:

• Cost per usable output is flat across a 400-fold change in concurrency, varying by 2% across the whole ramp with no trend, because the failure that dominates the denominator is a deterministic semantic error that load doesn’t touch.
• The strict arm is roughly 30% cheaper per usable output than the prompt-only baseline rather than more expensive. Enforcement costs a little in tokens and saves a third of all responses from dying at the parse rung, and the second effect is much larger than the first.

The five-step agent task gives the cleaner version of that comparison, because both arms held validity at 1.00 and the denominator drops out. Cost per usable output came to $0.001383 strict against $0.001832 prompt-only at concurrency 1, so strict was 25% cheaper, and the gap held at 10 and 50. With validity equal the whole difference is token count, and it runs the opposite way from the single-turn set, where dividing out the validity rates put prompt-only’s cost per request at about the same as strict’s. So enforcement wasn’t uniformly cheaper or uniformly more expensive in tokens. On the deeper task it was cheaper, and prompt-only was also 29% slower at concurrency 100, which points at the same cause. Splitting that into input and output tokens needs the per-request counts rather than the aggregate.

That comparison has a floor. Both arms carry the same deterministic arithmetic failure in the denominator, so neither figure is a cost-per-correct-output. They are cost-per-output-that-passed-this-harness’s-checks. And because the failing task fails identically on every retry, these are lower bounds wherever the failure is the deterministic one, which the rest of this section is about.

The cost that matters is latency. Every retry is an additional request against the same saturated GPU. At a 4% failure rate you’re offering 1.042 requests per valid output, and a retried request costs the user two full generations serially.

The tax lands on your percentiles rather than your bill. Above a 1% failure rate, retries are common enough to occupy the p99 tail, but prevalence alone doesn’t prove that every p99 request is a retry, because the single-attempt and retry latency distributions can overlap. At a 4% rate, retries make up four times the mass represented by the slowest percentile. Working that through the sustained ramp’s own latencies gives an illustrative bracket rather than a computed percentile, because quantiles don’t add, so the true figure needs the per-request latency distribution or a simulation. The range below brackets a retried request between two typical attempts, at 2 × p50, and two slow ones, at 2 × p99.

So a 4% failure rate that costs 4% in tokens puts a retried request at 1.3× to 2.0× the unretried p99 in this illustrative bracket. At 400 it converts a ten-second tail into a twenty-second one. The token tax is a rounding error and the latency tax is the actual bill. Two things push the real number toward the top of that bracket or past it:

• Repair-prompt retries resend the failed output as input, so their second attempt prefills more than their first and runs longer than 2×.
• Retries are correlated with load rather than spread evenly, so they arrive in the window where the single-attempt latency is already at the top of its own range.

Retry budgets have a failure mode of their own. The table assumes independent attempts, which is why residual failure falls off geometrically. Truncation failures aren’t independent. If the response needs more tokens than the budget allows, every retry at the same budget fails identically, forever. The geometric model says 1.042 attempts, while reality says the request never succeeds and you’ve tripled the load for nothing. Retry logic that doesn’t branch on termination metadata builds a retry storm out of a budget misconfiguration.

The ramp shows what that looks like when the assumption fails completely. One of its three templates failed deterministically at temperature 0, so retries reproduced the identical wrong total every time, and for that task class the residual after any number of attempts is 100% rather than the geometric figure. Working the mixture through on the actual request split rather than an idealized third, 134 requests succeeded on the first attempt and 66 burned all three, giving 332 attempts for 134 usable outputs, or $0.003865 per valid output, a 148% tax. The idealized exact-one-third version gives $0.003900 and 150%, and the difference comes from the 67/67/66 cycling rather than from a modelling choice. Either way it’s an order of magnitude past the 17.65% at the table’s 15% ceiling.

Those denominators are easy to mix. The 100% figure is the residual for the failing task class and the 33% is the residual across the mixture. Neither is a global failure rate, and both come from three deterministic templates rather than a sample. A retry budget prices independent failures, and a deterministic semantic error isn’t one.

The engineering payload is to assume failure and design accordingly. LLM Tool Calling with DigitalOcean AI Platform and Databases covers the single-request version of this guidance, and everything below is what changes once concurrency and multiple grammar backends enter the picture. A Simple Guide to Building AI Agents Correctly is where this section’s validation ladder belongs architecturally, inside the typed-tool orchestrator layer rather than the prompt.

The validation ladder, and what each rung costs

The ladder has four rungs, run in order, each catching what the one before it can’t:

1. Check termination metadata. Before touching the payload, read the provider’s stop signal, listed in the table under Truncation above. A response that was cut off should be rejected here and never reach the parser, because json.loads is a poor truncation detector. It only sees malformed JSON, and a repair layer sitting in front of it can make truncated output parse cleanly.
2. Parse. Run json.loads, which catches malformed output and extraction failures.
3. Schema-validate. This catches constraint-boundary failures, meaning the keywords your backend didn’t enforce.
4. Semantic-validate. Apply field-level business rules, which catch what nothing else can: empty required strings, arithmetic that doesn’t add up, dates in the wrong order, and IDs with the wrong prefix.

Run the ladder as application middleware rather than prompt hope, because no system-prompt instruction substitutes for any of it.

Rung 1 costs a dictionary lookup, so the cost question is about the other three. The figures below are measured per document on jsonschema 4.26.0 with Python 3.12.3. The benchmark runs a single timed loop per operation, so each is one measurement rather than a distribution.

The whole ladder lands between 66 and 316 microseconds. Set that against the generation it guards, since the burst ramp measured p50 latencies of 1.96 to 2.74 seconds. At the most expensive schema shape, validation costs one part in 6,200 of the request, and the ratio only improves under load, since the sustained ramp’s p50 reached 9.39 seconds at concurrency 400 while validation cost stays fixed.

Cost is the easy half. What each rung actually caught across the runs in this article is the half that decides whether you build it.

Those shares are properties of the configurations that produced them rather than failure rates you should expect. Each one is a task set built to provoke a specific rung, and three of the four are deterministic at temperature 0.

Across the 28,246 requests of the sustained strict ramp, rungs 1, 2 and 3 caught nothing whatsoever, and rung 4 caught every content failure that occurred. Termination metadata never fired because truncation was 0.00 at every level, parse never fired because the grammar makes a stray backtick unsamplable, and schema validation never fired because the keywords in play were all genuinely enforced.

On a pinned stack with well-supported keywords and an adequate budget, enforcement really does retire the first three rungs. That’s also the condition under which teams conclude the ladder is unnecessary and stop at the rung that has stopped catching anything. The rung that costs 77× to 104× less than schema validation, and gets skipped most often because nobody wrote the rules, was the only one doing work. That result inverts once any of those three conditions breaks, which is what the other rows in the table are. Each rung is idle until the day its precondition changes, and the change is usually a config edit somebody made for an unrelated reason.

Running the same script on jsonschema 3.2.0 with Python 3.10.12 and different hardware shows how portable those microseconds are. Correctness rates matched to four decimal places and the timings didn’t, with schema validation running 5.6× to 7.7× faster on the older stack. The parse column is the control, since json.loads never touches jsonschema. It ran 5.2× faster there against 6.6× for schema validation, so most of the gap is interpreter and hardware and only about 1.3× is plausibly the validator library. Either way, a µs validation figure without a version and hardware stamp isn’t a number.

There is no performance argument for skipping the cheapest rung. If you’re not running semantic validation, nobody wrote the rules, which is a backlog problem rather than a latency one.

Observability: what to log and alert on

Structured-output failures are cheap to detect and easy to miss, because the signals live in different places: termination metadata, backend-election state, and the validation ladder’s own rungs. This section collects the alerting advice scattered through the results above into one practice, the structured-output analog of the p99 monitoring Why your vLLM p99 latency blows up in production prescribes for latency.

Log six fields per request:

1. Termination status, read from the field rather than inferred from content. The Truncation table above gives the field name per provider, whether finish_reason, status, or stop_reason. It feeds the truncation-rate metric below, and it’s what separates a genuinely truncated response from a healthy null-content tool call.
2. Which backend got elected, if you’re on vLLM’s auto resolution. The engine logs nothing about which backend a process latched onto, so “which backend served this request” is undiscoverable after the fact unless you capture it yourself, once at startup or once per election.
3. The validation-ladder outcome per rung, rather than a single pass or fail. Parsed, schema-valid and semantic-valid are three different signals, and collapsing them into one boolean throws away the information that tells you which failure class you’re looking at.
4. One normalized outcome on every failed request, by a rule you’ve written down. A per-rung boolean says a request failed, while the normalized outcome says which fix applies, and the two aren’t the same field.
5. The raw completion alongside the parsed result, at debug level at minimum. Without it, a parser bug that mangled well-formed JSON and a model failure that produced genuinely malformed output look identical, and the retry branches below need that distinction to pick a branch at all.
6. Time-to-first-token and per-token decode time, rather than only end-to-end latency. A grammar-compiler cache too small for your working set leaves every rung of the ladder reporting success while both of those climb, as the cardinality results above measured. End-to-end latency moves too, but it moves for a dozen reasons. TTFT and per-token decode separate “the request waited” from “every token cost more,” and the second is the shape a cache problem takes. Neither number means anything absolute, so record them against a control run carrying a handful of schemas and watch the gap.

The fourth of those needs a deterministic, ordered rule, because failures overlap. A truncated response that also fails to parse is a truncation rather than a parser failure, since raising the budget is the fix and re-parsing isn’t. The measurements in this article take transport errors out first, before the taxonomy runs at all, because a request that produced no document has no structured-output category to be assigned. The rest then order as truncation by termination metadata, then request-time schema rejection, then a parsed document that violates a schema you asked to be enforced, then parse failure on a complete response, then semantic. That’s one defensible ordering rather than the only one. What matters is that it’s fixed, that every failure lands in exactly one bucket, and that two categories which take opposite fixes get separate labels. A constraint the backend silently ignored and a schema the backend refused outright are both constraint-boundary failures, and confusing them wastes an incident.

With those six fields logged, alert on four rates rather than one:

1. Validity rate, the ladder’s overall pass rate, which is the one most dashboards already have.
2. Truncation rate, tracked separately from validity rate, which catches what a validity-only dashboard hides. Enforcement can hold on every request that completes while a large share of attempts never finish at all, and conflating “the constraint failed” with “the constraint held and the budget didn’t” sends you to fix the wrong thing.
3. Zero-token responses, tracked apart from malformed JSON, which are indistinguishable from a hung request unless retained status, finish metadata and usage counts are all logged.
4. Transport-failure rate, counted apart from anything schema-shaped, because folded into a single validity number a client-side drop reads exactly like structured output degrading under concurrency. That was the misattribution this study walked into itself. Alert on the rate rather than on the load level you expect it at, since queue depth predicted neither its onset nor its absence. When this one is what’s moving, suspect your own client before the engine.

None of this needs new tooling. These are the same fields the rest of this article already asks you to check, captured as a standing practice instead of read once during an incident.

Retry design that doesn’t make things worse

Branch on the failure category before retrying. Four categories take four different branches:

1. Truncation, detected from the provider’s termination metadata, meaning finish_reason == "length" on Chat Completions, status == "incomplete" on Responses, or stop_reason == "max_tokens" on Anthropic. Raise the output budget and retry, or fail the request. Don’t retry at the same budget, since that’s the retry-storm mechanism described above. Don’t repair either, because the truncation results show repair recovering well-formedness far more readily than content.
2. Parse failure with a complete response, which could come from either side. Check the raw completion first. Well-formed raw text with a garbled parse result is an extraction bug and retrying won’t help, while malformed raw text is model behavior, and a markdown code fence is the most common shape of it.
3. Schema-invalid but parseable, which is where repair-prompt retries earn their keep, since resending with the validator error attached is more informative than blind regeneration. It isn’t cheaper, since the repair path carries the failed output plus the error as extra input on every attempt after the first. Choose it for information content rather than price, and don’t assume it converges faster without measuring that on your workload.
4. Semantically invalid, where you should retry but change something. The same prompt and the same schema at temperature 0 gives you the same wrong answer, so re-ask with the specific rule violation quoted back.

Keep the retry budget at 2 or 3. Residual failure after three attempts is below 0.01% at around a 4% independent per-attempt failure rate, which isn’t a generic number. At 15% it’s 0.34%, and if attempts aren’t independent it doesn’t apply at all.

Preserve prefix stability across all four branches. Tool and schema definitions are exactly the long stable prefix that prompt caching rewards. DO’s measurements show a 3,110-token prompt going from 462 ms TTFT cold to 39 ms warm, a 92% reduction, and an agent session with a stable 2,080-token prefix holding a 98.7% cache hit rate that collapses to 0.7% when the prefix is disturbed. A repair-prompt retry that appends the error after the stable prefix keeps the cache. One that rewrites the system prompt to add “please be more careful this time” throws away the hit and adds latency to a request that’s already late.

Schema design as reliability engineering

Treat the schema as a control surface. Four choices measurably change your failure rate, and one recommendation from the engines’ own documentation sits underneath all of them: align the prompt with the enforced schema, since enforcement fights the model when the prompt doesn’t cooperate.

1. Prefer enums to free strings. An enum is enforced by the mask at sampling time, so out-of-set values are unsamplable, where a free-form string field constrains nothing beyond the quotes. This narrows the failure space rather than eliminating it, because the model can still pick the wrong valid member and no constraint will tell you it did.
2. Fail loud on purpose. The truncation results above held the array-of-objects schema’s validity at 0.00 through nine deciles of completion, against a flat schema that reached 1.00 by the 60% mark, because minItems and required per-record fields reject a half-written document instead of accepting it. Required fields and minItems are the difference between a truncation you catch and one you ship.
3. Consider ordering fields so reasoning precedes conclusions. Putting a reasoning field before answer is a cheap thing to try, on the hypothesis behind Tam et al.'s ordering correlation. Two caveats carry over from the semantic-failures results above. Key-order preservation is an OpenAI guarantee rather than a universal one, and the recovery effect is inferred rather than demonstrated. Measure it on your workload instead of assuming it.
4. Size the budget in target-model tokens, from realistic conforming instances. Generate a handful of outputs that actually satisfy your schema, tokenize them with the model’s own tokenizer, take the upper end, and add reasoning headroom. Serialized character length is a weak proxy, because tokenizers split JSON punctuation, field names and numbers unevenly, so two documents of equal length can differ substantially in token count. A schema with unbounded strings or unbounded arrays has no finite worst case at all, which is itself the signal, so add maxLength and maxItems where you can and give the budget question an answer. This number moves your failure rate more than anything else on the list, and it’s usually a copy-pasted 512.

Four modes, in rough order of how much you should trust them:

1. Prompt-only JSON isn’t for unattended production agents, and the reason is variance rather than a bad average. The same baseline arm parsed at 0.67 on the single-turn set, losing a third of responses to a markdown fence, and at 1.00 across 3,000 model calls on the five-step task. Nothing about the two tasks predicted which would happen, and the failing one failed identically every time, so you can’t sample your way to confidence either. StructuredRAG (Shorten et al., arXiv:2408.11061, preprint) reports the same shape at larger scale, with an 82.55% average success rate across 24 prompt-only experiments, “ranging from 0 to 100%.” It’s acceptable only for human-reviewed, low-volume flows.
2. Strict constrained decoding is the default for agent pipelines, and it delivers exactly what it claims. Excluding client transport drops, schema validity among returned responses held at 1.00 across the whole sustained ramp, including at concurrency 400 with a real queue behind the batch, and it cost about 30% less per usable output than the prompt-only baseline because it stops a third of responses from dying at the parse rung. You now own three caveats: budget sizing for truncation, per-engine verification of which schema keywords are actually enforced, and semantic validation regardless. The third is where the strict arm’s only content failure landed, and the relocation result above says that failure was already present in the baseline, failing at a different rung.
3. Function calling with strict tool schemas is for when the output is an action. Set strict: true explicitly rather than relying on tool_choice="auto", and verify the echoed strict field on OpenAI’s Responses API. Build Real-Time AI Agents with DigitalOcean AI Platform and Serverless Functions covers the function-schema basics on this same platform, and this article picks up where it left off, at the point where the parser rather than the schema decides what your application sees.
4. Validation and retry belong in every mode. Two retries take a 4% first-attempt failure rate to a residual 0.006%, per the cost table above, for a 4% token tax and well under a millisecond of validation per document. That’s cheap reliability, with the caveat the ramp exposed: retries do nothing for a deterministic semantic failure.

Three workloads pull that in different directions:

• High-concurrency agent fleets want strict mode, retries branched on termination metadata before anything else, and logging and alerting per Observability above. Check your offered concurrency against max_num_seqs before concluding anything about server contention, and monitor transport failures separately because queue depth predicts neither their onset nor their absence. If you serve many distinct schemas, size VLLM_XGRAMMAR_CACHE_MB against your real working set and verify it by comparing TTFT and per-token decode against a run carrying a handful of schemas, since the penalty tracks working-set bytes against cache capacity rather than schema count.
• Batch extraction is the place to spend on semantic validation, since latency doesn’t bind. Use fail-loud schemas with minItems and full required lists.
• Latency-sensitive interactive tools want flat schemas, small enums and generous budgets. A retry costs a full generation, so the goal is a first-attempt success rate high enough that retries stay rare.

The LLM Inference Trilemma covers the SLO and batch-size framing this per-workload guidance sits inside, and batch-size choices interact directly with the truncation-budget discussion above.

Structured output reliability is a systems property rather than a model feature. It emerges from the interaction of decoding constraints, token budgets, which backend compiled your grammar, and extraction parsing, and three of those four are invisible if all you inspect is the model’s response.

Strict mode delivered the structural guarantee it advertises. Schema validity among returned responses held at 1.00 from concurrency 1 to 400 under sustained load, and at 1.00 over every request at 400 once the client’s connection pool left the path, with 266 still queued and p50 near ten seconds. Contention moved p99 by roughly fivefold and validity not at all, and the other load hypotheses went the same way. Truncation is fixed by the task rather than concurrency, constraint-boundary failures are a constant offset identical at every level, and five turns of depth introduced no new failure mode. All of that describes one pinned stack. The same requests against a managed endpoint slid from 1.00 to roughly 0.85, so whether load breaks validity depends on where you run, and on a managed stack you can’t see which part of it decides.

What enforcement doesn’t do is make the checks most teams run sufficient. The only content failure anywhere in this study was an arithmetic error inside a schema-valid document, a field-level rule was the only rung that caught it, and on a matched two-arm run that failure was already present in the unenforced baseline, failing at the parse rung instead. Two more failures have the same silent shape. Repair recovers well-formedness far more readily than content, turning a detectable truncation into an undetectable substitution, and a compiler cache too small for your working set costs 2.5 times the TTFT while every rung of the ladder reports success.

Constrained decoding is the right default, and it’s one check of four. Build the other three, so that you read the provider’s termination metadata before you parse anything, validate against the schema, and run field-level business rules on whatever survives. Then record the engine, backend, grammar-library and validator versions with every number you report.

• Why your vLLM p99 latency blows up in production documents the same contention this ramp measured, from the latency side instead of the validity side.
• Token Economics Across Traffic Profiles on Dedicated GPUs is the cost framework behind the cost-per-valid-output tables.
• Multi-Model Routing Is an Infrastructure Decision, Not a Feature carries a section, “Misrouted requests: the silent failure,” that is this article’s thesis one layer up: a request answered by the wrong model while reporting success.
• Multi-Model API Cost Governance with the Inference Router reports a GPT-5 call at max_completion_tokens: 1024 returning "content": null with "finish_reason": "length" because reasoning consumed the budget, which is the mechanism from Truncation above. Its fix was to raise the budget, which is the branch this article’s retry design prescribes rather than a repair.

• Inference pricing
• GPU Droplets pricing
• Why your vLLM p99 latency blows up in production
• How Does Prompt Caching Work and When Does It Actually Cut LLM Costs?
• Fine-tuning the LLM Ornith 9b on a single H200 GPU Droplet
• Metrics that Matter with Serverless Inference
• Token Economics Across Traffic Profiles on Dedicated GPUs
• LLM Tool Calling with DigitalOcean AI Platform and Databases
• A Simple Guide to Building AI Agents Correctly
• What’s New on DigitalOcean’s Inference Engine
• vLLM 1-Click Model (Marketplace)
• The LLM Inference Trilemma
• Build Real-Time AI Agents with DigitalOcean AI Platform and Serverless Functions
• Multi-Model Routing Is an Infrastructure Decision, Not a Feature
• Multi-Model API Cost Governance with the Inference Router