Postmortem: A Cold Start Looked Like a Refusal, and the Patch Queue Stopped

The short version

A gate that reviews AI-generated patches must separate three outcomes.

They are: the patch passed, the patch failed review, and nobody knows yet.

Our gate only had two. A 502 from a cold upstream became blocked.

That is a verdict. It is not a fact. The queue stopped for the wrong reason.

This post is a postmortem of that failure class. It covers the timeline, the contributing factors, the reproduction harness, and the durable fix.

All timestamps below come from a local reproduction harness. No production numbers are claimed.

Scope and method

The failure mode was rebuilt in a small sandbox. One fake reviewer server. One gate script. One queue of ten patches.

The harness is included in full. Anyone can rerun it and watch the same misclassification happen.

Timeline

The harness used a fake reviewer that sleeps, then returns 502 upstream connect error. The gate called it four seconds after boot.

The last row is the tell. Nothing changed except time.

Contributing factors

1. A health check that proved liveness, not readiness. The port answered TCP. The upstream did not answer anything. The gate treated those as the same signal.
2. Error classification by status family, not by meaning. Any non-200 became a rejection. Infrastructure errors and model judgements shared one branch.
3. An empty-body branch with the same bug. A blank body also became blocked. Two paths, one wrong default.
4. No idempotency key. Retries were impossible to add safely, so nobody added them.
5. A two-value verdict type. pass and blocked cannot express "retry me". The type forced the wrong default.

The fifth factor is the root cause. The first four are how it got through review.

The reproduction harness

Start with the fake reviewer. It fails the way a cold upstream fails.

# harness/fake_reviewer.py from http.server import BaseHTTPRequestHandler, HTTPServer import json, sys, time MODE = sys.argv[1] if len(sys.argv) > 1 else "cold" class Handler(BaseHTTPRequestHandler): def do_POST(self): n = int(self.headers.get("content-length", 0)) self.rfile.read(n) if MODE == "cold": time.sleep(2) self.send_response(502) self.end_headers() self.wfile.write(b"upstream connect error") return self.send_response(200) self.send_header("content-type", "application/json") self.end_headers() self.wfile.write(json.dumps({"verdict": "pass"}).encode()) def log_message(self, *args): pass HTTPServer(("127.0.0.1", 8081), Handler).serve_forever() 

Now the buggy gate. It is twelve lines and it looks reasonable.

# harness/gate_buggy.py import json, sys, urllib.request def naive_verdict(status: int, body: str) -> str: if status != 200: return "blocked" # infrastructure became a judgement  if not body.strip(): return "blocked" # the same mistake, second path  return json.loads(body)["verdict"] status, body = 502, "upstream connect error" print(naive_verdict(status, body), file=sys.stderr) 

Run both. The gate says blocked. Rerun four seconds later and it says pass.

python harness/fake_reviewer.py cold & sleep 1 && python harness/gate_buggy.py sleep 6 && python harness/gate_buggy.py 

The durable fix

Three changes. Each one removes a way for the gate to guess.

1. A tri-state verdict type

unknown is a first-class outcome. It is not a synonym for failure.

from enum import Enum class Verdict(str, Enum): PASS = "pass" FAIL = "fail" UNKNOWN = "unknown" INFRA = {408, 425, 429, 500, 502, 503, 504} def classify(status: int, body: str) -> Verdict: if status in INFRA: return Verdict.UNKNOWN if status in (401, 403): return Verdict.UNKNOWN # our credential problem, not a review  if status >= 400: return Verdict.FAIL # the reviewer rejected the request itself  try: payload = json.loads(body or "{}") except json.JSONDecodeError: return Verdict.UNKNOWN v = payload.get("verdict") return Verdict(v) if v in {"pass", "fail"} else Verdict.UNKNOWN 

2. Retries with an idempotency key

The key is derived from the patch and the model tag. Two retries of the same patch collapse into one recorded review.

import hashlib, random, time def request_key(patch: str, model_tag: str) -> str: return hashlib.sha256(f"{model_tag}:{patch}".encode()).hexdigest()[:16] def review(call, patch: str, model_tag: str, attempts: int = 3): key = request_key(patch, model_tag) last = Verdict.UNKNOWN for i in range(attempts): status, body = call(patch, idempotency_key=key) last = classify(status, body) if last is not Verdict.UNKNOWN: return last time.sleep((2 ** i) * 0.5 + random.random() * 0.3) return last 

3. A gate policy that never guesses

Unknown is not green. It is a hold, and a hold has an owner.

def gate(v: Verdict) -> str: return {Verdict.PASS: "green", Verdict.FAIL: "red"}.get(v, "hold") 

4. A readiness probe, not a liveness probe

Probe the review path with a cheap payload. Require a valid JSON verdict before draining.

curl -sS -o /tmp/warm.json -w '%{http_code} %{time_total}\n' \ -X POST "$REVIEWER_URL/review" \ -H 'content-type: application/json' \ -d '{"patch":"warmup","warmup":true}' python -c 'import json;print(json.load(open("/tmp/warm.json"))["verdict"])' 

Verification plan

Add these tests before shipping the fix. Each one maps to a contributing factor.

1. 502 at t=1s returns unknown, never blocked.
2. Empty body with status 200 returns unknown.
3. Truncated JSON with status 200 returns unknown.
4. 401 returns unknown and pages the operator.
5. Three consecutive unknown results produce hold, not green.
6. The same patch retried twice writes one review record, not two.
7. Queue drain waits for one successful readiness probe.

A gate that fails these tests is still guessing.

Where free access fits, and what I used

This kind of harness is cheap to run. The reviewer tier only needs a model call. The gate runner only needs one small process.

For the sandbox, the reviewer used MonkeyCode's free model access. The gate runner sat on the free server option.

Both kept the experiment at zero cost while the failure mode was being chased.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest notes. Free tiers change, so verify current limits on the project page before you size a queue. And a free runner is still a runner. It can cold-start, and it can be evicted mid-request.

That is exactly why the tri-state fix matters more on free infrastructure, not less.

Limitations

• The harness uses a synthetic failure. Real cold starts are less polite.
• unknown adds operator load. Every hold needs a human with a deadline.
• Idempotency keys help only if the reviewer honours them.
• This does not make a weak reviewer correct. It only stops it from lying.

Who should not use this

Do not adopt the tri-state gate if any of these are true.

• Nobody is on call to clear hold rows.
• The pipeline treats any non-green as a hard stop with no owner.
• Patches are low risk and volume is very high on a shared queue.

In those cases, a two-value gate with a manual rerun policy is the smaller evil.

Takeaways

Separate infrastructure failure from review outcome. Make unknown a real state. Give it an owner.

Then a cold start is a delay, not a verdict. And the queue keeps moving.

If you want to rerun this harness, start from the project's current docs and check the free-tier terms yourself before you commit a queue to them.