The Prompt Canary: Catching Silent Input Truncation Before Your Nightly Job Ships Bad Output

Why does a nightly summarisation job keep producing summaries that sound confident and completely ignore the last third of every document? Nothing crashes, no retry fires, the queue drains by morning, and the only symptom is a human noticing that paragraph nine never made it into the output. I spent a 48-hour window chasing that exact class of bug, and the useful part was not the fix — it was the probe I now run before the job is allowed to leave my laptop.

Here is the reframe that made it tractable. Most teams instrument the model's output and never instrument the input the model actually received. If a gateway trims a long prompt, drops a system message, or silently truncates a conversation history, your logs still show the request you sent, not the request that was served. So the question becomes: how do you prove the payload arrived intact, using only the endpoint you already have?

The failure mode I was not measuring

Silent truncation is nasty because it degrades quality without degrading reliability. Your error rate stays at zero, your p95 latency looks healthy, and your structured JSON still parses — it is just answering a shorter question than the one you asked. Retries make it worse, because every retry re-sends the same oversized payload and re-truncates it in the same place.

There are three distinct variants worth separating, since each one needs a different remedy:

• Tail truncation: the end of the prompt is cut, so your final instructions or closing context vanish.
• Head or system loss: the system message is dropped or summarised away, so tone and output contract drift.
• Middle truncation: the model sees the start and the end but loses the middle, which is the hardest to notice by eye.

A tokenizer library tells you what you think the budget is. It does not tell you what the serving path did. Those two numbers diverge exactly when you need them most.

The canary: two random tokens and a JSON contract

The probe is deliberately boring. Put one random token in the system message and one at the very end of the user message, then ask the model to echo both back verbatim as JSON. If the head canary comes back and the tail does not, you have tail truncation. If the system canary vanishes, something ate your system prompt.

import json, os, secrets, urllib.request BASE = os.environ["MODEL_BASE_URL"].rstrip("/") KEY = os.environ.get("MODEL_API_KEY", "") MODEL = os.environ["MODEL_NAME"] def canary() -> str: return "CNRY-" + secrets.token_hex(4).upper() def probe(body: str, max_tokens: int = 64) -> dict: head, tail = canary(), canary() payload = { "model": MODEL, "temperature": 0, "max_tokens": max_tokens, "messages": [ {"role": "system", "content": f"Echo both canaries verbatim. head_canary={head}"}, {"role": "user", "content": f"{body}\n\ntail_canary={tail}\nReply with JSON only: {{\"head\":\"...\",\"tail\":\"...\"}}"}, ], } req = urllib.request.Request( f"{BASE}/chat/completions", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json", **(({"Authorization": f"Bearer {KEY}"}) if KEY else {})}, ) with urllib.request.urlopen(req, timeout=90) as r: text = json.loads(r.read())["choices"][0]["message"]["content"] out = json.loads(text) # raises if the contract was ignored  got_head, got_tail = out.get("head"), out.get("tail") if got_head == head and got_tail == tail: verdict = "OK" elif got_tail == tail: verdict = "SYSTEM_LOST" elif got_head == head: verdict = "TAIL_CUT" else: verdict = "BOTH_MISSING" return {"verdict": verdict, "head": got_head, "tail": got_tail, "body_chars": len(body)} 

Set max_tokens low on purpose. You are paying for four tokens of echo, not for an answer, so the probe stays cheap enough to run on every template change.

Calibrating the budget by doubling, not by guessing

Once the probe works on a short body, I find the effective ceiling empirically. Start with a small fixture, double the padding, and stop at the first non-OK verdict. The transition point is the number I actually care about, and it is a property of the serving path, not of my tokenizer version.

def sweep(template: str, start: int = 2000, limit: int = 400_000) -> list[dict]: results, size = [], start while size <= limit: body = template + (" The quick brown fox jumps over the lazy log line." * (size // 52)) row = probe(body) row["target_chars"] = size results.append(row) print(row["target_chars"], row["verdict"]) if row["verdict"] != "OK": break # first failure: back off and inspect the boundary  size *= 2 return results 

Run it twice back to back and compare. If the verdict flips between identical runs at the same size, your endpoint is load-balanced across heterogeneous backends, and you should pin the boundary at the lowest observed ceiling rather than the average.

This is the stage where a free tier genuinely earns its place. MonkeyCode's free model access and free server option are the operator-supplied reasons I could run a doubling sweep repeatedly without thinking about spend, which matters because the sweep is intentionally wasteful when the endpoint is a paid one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have a cheap staging endpoint, the same harness works there unchanged.

The 48-hour field plan

I break the two days into four blocks, and the last block is the only one that touches production code:

1. Hours 0–2 — baseline. Run probe() on a 500-character fixture. If this fails, stop: your problem is the request contract, not truncation.
2. Hours 2–10 — sweep and record. Run the doubling sweep, write the first-failure size and the verdict to a file keyed by a hash of the prompt template.
3. Hours 10–36 — replay real inputs. Feed the harness ten real payloads from your logs at their true sizes, still with max_tokens low, and record verdicts. This is where hairline cases show up, usually as TAIL_CUT on documents that are only marginally oversized.
4. Hours 36–48 — wire the guard. Add a preflight that re-runs one probe whenever the template hash changes, and refuse to enqueue a batch whose payload exceeds the recorded ceiling.

What to do with each verdict

A verdict is only useful if it maps to an action you are willing to take at 2 a.m. Mine looks like this:

Notice that only one row is really about tokens. Two of the five are contract bugs that a bigger budget will never fix, which is precisely why guessing from a tokenizer count wastes so much time.

Limitations, and who should skip this

Be honest about the scope of the canary. It proves that a payload arrived intact and that the output contract survived — it says nothing about answer quality, hallucination rate, or latency. It also cannot reliably detect middle truncation, because both sentinels can survive while the centre of the prompt is discarded.

You should probably skip this approach if any of the following is true:

• Your payloads are consistently far below the ceiling, so there is no truncation to catch.
• You already control the serving stack and can read the tokenizer and truncation config directly.
• Your endpoint strips or rewrites randomised tokens for safety reasons, which makes verdicts noisy rather than informative.
• You need quality regression testing, because that is an eval harness, not a canary.

If any of those apply, spend your two days on a golden-set eval instead, and let this probe be a five-minute smoke test rather than a project.

What I would repeat

Two things earned their place. First, running the sweep before touching application code, because it produced a number I could guard on instead of a suspicion. Second, keying the recorded ceiling to a hash of the prompt template, so a careless edit invalidates the measurement instead of silently outliving it.

What I would change is ordering: next time I would write the SYSTEM_LOST branch first, since a dropped system message looks exactly like a model that has stopped following instructions, and I burned hours blaming the model for my own missing payload.

What does your job do when the last instruction in a prompt quietly never arrives? If you cannot answer from a log line you already keep, a two-token canary is a cheap way to find out before a reader does.