Picture an internal expense agent. Someone asks whether a flight change fee is reimbursable. Retrieval returns zero chunks — the policy PDF failed to parse — and the agent answers "Yes, covered under the standard travel policy", then advances the request to approved. Nobody notices until the end of the month.
The postmortem usually stops at the word hallucination. That ends the investigation one step too early. In most of these systems the system prompt already contains some version of if you don't know, say you don't know. The rule was written. It just wasn't a decision the system could make: there was no state for insufficient_context, no transition out of pending, and no place to record whether the rule was consulted on this input. A language model asked to answer a question with an empty context produces a plausible continuation. That is not a malfunction; it is what the component does.
"I don't know" is three different things
A string. A sentence in the system prompt. Cheap to add, impossible to audit: the output may contain the phrase while the state machine still advances the request.
A state. insufficient_context exists in the orchestrator, with a defined transition and an owner. Now an answer can be refused without relying on prose. But you still cannot tell, after the fact, whether the condition was evaluated on a run that should have triggered it.
An event. constraint_evaluated is written into the run record with the input, the criterion and the fallback reason. Now "did this rule fire tonight?" has a queryable answer.
Only the third version is testable. The first two are how a team ends up adding the same sentence to the prompt three times and wondering why nothing changed.
The boundary belongs to the deterministic backbone
Under hybrid orchestration, a deterministic backbone owns state, retries, authorization, audit trails and rollback; a model is invoked only at nodes requiring semantic judgement. The interesting word there is nodes. Whether there is enough evidence to answer is not a semantic judgement about the answer — it is a check on fields that already exist before the prompt is assembled: retrieved_chunks, schema_valid, scope_match, attachment_parsed.
A rule of thumb we've found useful: if a decision can be made entirely from values present before the prompt is assembled, it belongs in code. Asking the model that generates the answer to also certify the evidence makes the same component both author and reviewer. The model can still misread a policy it can read — that is a real failure, but it lives at a different node and needs a different test.
def answer_policy_question(run): evidence = retrieve(run.question) if not evidence or not evidence.schema_valid: run.record(constraint="missing_context -> insufficient_context", constraint_evaluated=True, fallback_reason="empty_retrieval", state_transition="pending -> blocked", model_called=False) return AskForClarification(reason="empty_retrieval") return model.answer(question=run.question, evidence=evidence) The output contract has to be typed as well, or the string comes back through another door. status is a two-value enum, and answer is null whenever the status is not answered:
{ "status": "insufficient_context", "answer": null, "evidence_ids": [], "fallback_reason": "empty_retrieval" } An illustrative run record for that path would look like this:
{ "run_id": "r-184", "node": "answer_policy_question", "input": {"question": "Is the change fee reimbursable?", "retrieved_chunks": 0, "schema_valid": true}, "constraint": "missing_context -> insufficient_context, never infer", "constraint_evaluated": true, "fallback_reason": "empty_retrieval", "state_transition": "pending -> blocked", "model_called": false } model_called: false is the whole point. It is an assertion you can write against a replay, not a hope about tone. One more detail worth keeping: the fallback belongs in the structured prompt contract as a hard constraint and in code as a transition. If it exists only in the contract, you are back at version one with nicer formatting.
Nine cases, not one final-answer score
White-box evaluation means capturing per-node inputs, outputs and acceptance criteria into replayable test sets, instead of judging the final answer by feel. Three test sets, kept separate: positive / boundary / follow-up. For this one fallback, three cases each:
- Positive: grounded context, a clear answer, and the fallback must not fire.
- Boundary: zero retrieved chunks; a malformed tool response; a question outside the policy's scope. Each must end at
insufficient_contextwith the rightfallback_reason. - Follow-up: the user supplies the missing document; the user rephrases after a refusal; the user asks again after context was repaired. Same run, same candidate, no state leak.
Nine cases, two assertions each — the state transition and constraint_evaluated — give eighteen checks that all fail on the same line when the rule is only prose. Assert both directions. A fallback that fires on grounded context is a new failure mode, not a safety improvement, and a suite that only rewards refusals will happily produce an agent that refuses everything.
The same shape covers the other fallbacks: authorization denied, rollback, retry exhausted. Once every fallback is an owned state with an evaluation event, coverage becomes a count you can diff between deploys — declared fallbacks, fallbacks with a case that fires them, fallbacks with a case that must not.
What this costs, and what it doesn't buy
There is a cost consequence, but it runs downstream of the architecture. When insufficient_context is an owned state, you can count the runs that ended there and decide whether to fix parsing, extend the corpus, or loosen the rule. Cost stops being a number attached to "the agent was slow" and becomes a number attached to a named state. Useful — but not the reason to build the boundary, and framing it as a savings story tends to produce the wrong design.
Scope and limits, because this argument is easy to oversell:
- It does not fix semantic errors on genuinely ambiguous input, knowledge missing from the corpus, or requirements that were wrong to begin with. A perfect backbone around a bad question still produces a bad answer.
- Deterministic delivery here means controllability, auditability and fallbacks at the architecture level — explicitly not a promise that the model will always be right.
- It can add a clarification turn, and it introduces over-refusal as a failure mode you now have to test.
- For a throwaway prototype where a wrong answer costs nothing and nobody has to explain it later, a probabilistic black box is the cheaper and more honest choice. The backbone earns its keep when failures must be attributed, outputs reach external users, or an audit trail matters.
The question I'll leave you with
In your stack tonight, is "I don't know" a sentence in a prompt, a state the orchestrator owns, or an event you can replay to the exact input and criterion that should have stopped the call? If a policy document fails to parse at 2 a.m., which line of the run record would show that the rule was evaluated — and would the request still have advanced?
No tool to sell, no link to click. Just this: a fallback that only exists in prose is a fallback you have not built yet.
