Valid JSON, Wrong Order: Five Structured-Output Myths and a Three-Gate Replay Test

A pipeline of mine broke last week. Not on a timeout. Not on a 500.

It broke because the model returned 'quantity': '3'. A string. Twelve downstream orders went out with a quantity of zero.

So let me ask the boring question. Would your tests have caught that? Mine did not, and I wrote them.

Here are five claims I keep hearing about structured output. Each one comes with evidence you can collect yourself, and a corrected mental model.

Myth 1: JSON mode means my schema is enforced

No. It means the bytes parse as JSON. That is a syntax promise, not a contract.

I keep seeing this confusion in reviews. Someone enables a JSON flag and deletes the validator. Type drift then walks straight into the database.

Evidence you can collect in ten minutes: save 50 raw responses, then run them through json.loads. Count the parses. Then run the parses through a real schema validator. The two numbers will differ.

Corrected model: JSON mode protects the parser. Your schema protects the domain.

Myth 2: If it parses, the model was right

Parsing is the cheapest gate. It is also the least informative.

The output can parse, match the schema, and still be business-wrong. Totals that do not add up. A status the state machine never allows. An order id from another tenant.

Corrected model: validation has three layers, not one.

1. Gate A: does it parse?
2. Gate B: does it match the schema?
3. Gate C: does it survive your cross-field rules?

Most teams ship gate A and call it done. That is where my string '3' came from.

Myth 3: Retrying fixes bad output

Retrying with the same prompt is a coin flip. Sometimes it lands. You learn nothing either way.

Ask yourself what the retry actually changed. If the answer is nothing, you bought variance, not reliability.

Before you add a retry loop, measure your gate failure rate. If gate B fails 4% of the time, a retry hides it. It does not remove it. And you still cannot explain the 4%.

Corrected model: retries are a mitigation. The gate is the fix.

Myth 4: Strict mode enforces my constraints

Strict modes vary by provider. Some enforce types well. Fewer enforce regex patterns, ranges, or cross-field arithmetic.

Can your validator tell the difference between absent and null? Does it reject booleans where you expect integers? In Python, isinstance(True, int) is true. That bites people.

Corrected model: treat generated constraints as hints. Treat your validator as law.

Myth 5: Schema drift is a model problem

It is a contract problem. You version your API. Version your output schema too.

When a prompt changes and the field list changes with it, you have a silent breaking change. Nothing alerts. Nothing fails loudly.

Corrected model: the schema lives in version control, next to the prompt. It has a version number. The gate logs that version with every rejection.

The three-gate check

Here is the gate I use. It is a template, not a benchmark. I have run it on my own traffic; your failure mix will differ.

# gate.py import json from collections import Counter from jsonschema import Draft202012Validator SCHEMA = { '$schema': 'https://json-schema.org/draft/2020-12/schema', 'type': 'object', 'required': ['order_id', 'quantity', 'unit_cents', 'total_cents', 'status'], 'additionalProperties': False, 'properties': { 'order_id': {'type': 'string', 'pattern': '^ord_[0-9a-f]{8}$'}, 'quantity': {'type': 'integer', 'minimum': 1, 'maximum': 999}, 'unit_cents': {'type': 'integer', 'minimum': 0}, 'total_cents': {'type': 'integer', 'minimum': 0}, 'status': {'type': 'string', 'enum': ['new', 'paid', 'shipped']}, }, } VALIDATOR = Draft202012Validator(SCHEMA) COUNT = Counter() def gate_a(raw): try: return json.loads(raw), None except json.JSONDecodeError as e: COUNT['gateA'] += 1 return None, 'gateA:not-json@%d' % e.pos def gate_b(obj): errors = sorted(VALIDATOR.iter_errors(obj), key=lambda e: list(e.path)) if errors: path = '/'.join(str(p) for p in errors[0].path) or '$' COUNT['gateB'] += 1 return None, 'gateB:schema:' + path return obj, None def gate_c(obj): if obj['total_cents'] != obj['quantity'] * obj['unit_cents']: COUNT['gateC:arithmetic'] += 1 return None, 'gateC:total-mismatch' return obj, None 

Three functions. Three failure counters. That is the whole idea.

One nuance: coerce before you reject. '3' can become 3. But log the coercion. A rising coercion rate is a prompt regression, and you want to see it.

The replay harness

You do not need a live model to test this. Save raw responses, then replay them.

# replay.py — expects JSONL where each line is {'content': '<raw model output>'} import json, sys from gate import gate_a, gate_b, gate_c, COUNT kept = 0 with open(sys.argv[1], encoding='utf-8') as fh: for line in fh: raw = json.loads(line)['content'] obj, err = gate_a(raw) if err: continue obj, err = gate_b(obj) if err: continue obj, err = gate_c(obj) if err: continue kept += 1 print('kept', kept, dict(COUNT)) 

Run it like this:

python -m venv .venv && . .venv/bin/activate pip install jsonschema python replay.py saved_responses.jsonl 

Now the output is a number you can argue about. Not a vibe.

Decision table

That last row matters. Stop asking the model to do arithmetic. Ask for quantity and unit_cents. Multiply in code.

Running the gate next to the model

Here is the practical part. The gate should sit in the same process as the model call. Network hops cost latency. Latency costs retries.

That is why I keep the gate script on the same box as the model call. If you are trying this with free model access, MonkeyCode describes a free server option that lets you co-locate them. The offer described to me is free model access with a token allowance reported as 10 million tokens, plus that free server option. Quotas change, so check the live page before you plan around a number.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free tier is fine for the replay loop. It is a poor place for a p99 latency claim. Do not confuse the two.

Limitations you should hear

• The gate catches contract violations, not hallucinations. A field can be valid and still wrong.
• Gate C is domain code. It will never be generic, and it should not be.
• My replay numbers are not published here because they are my traffic. Replay your own.
• Schema libraries differ on format support. Test the validator, not just the model.

Who should not use this

Skip the three-gate design if your SDK already enforces a typed contract end to end. Skip it if your output is prose for humans, with no downstream consumer. Skip it if you cannot store raw responses, because then replay is impossible.

Also skip it if you plan to hide gate failures behind retries. That defeats the point.

A five-minute checklist

1. Save 50 raw model responses today.
2. Write the schema. Add additionalProperties: False.
3. Run the replay. Write down three numbers.
4. Add one gate C rule. Just one.
5. Re-run. Compare.

So what does your schema actually guarantee? If the answer is only that the bytes parse, you have a gate A pipeline. That is a start, not a contract.