Postmortem Drill: Four of Five Files Applied, and the Gate Called It Green

A patch gate can pass a change that was only partly applied. The bug hides in a warning where an abort belongs. This postmortem covers one reproducible shape of that bug. It includes the timeline, four contributing factors, and an all-or-nothing fix.

The drill below is synthetic. It is not a production outage. Every step is reproducible with the small harness in this post.

The Failure In One Paragraph

A model returns a five-file diff. The gate splits it by file. It applies each file in a loop. File five fails on stale context. The loop logs a warning and continues.

Tests then run on a tree with four of five files present. Those tests pass, because they never touch the missing file. The verdict is recorded as pass. The stored artifact hash covers all five files.

The gate validated a subset. It reported on the superset. That gap is the entire incident.

Timeline (Synthetic Drill)

Clock	Event	Gate state
t+00:00	Batch job submits a five-file diff for a narrow refactor	queued
t+00:04	Splitter emits five independent chunks, one per file	running
t+00:11	Chunks 1 to 4 apply cleanly	running
t+00:13	Chunk 5 hits a context mismatch, exits non-zero	warning logged
t+00:13	Loop continues, no rollback	running
t+00:14	Suites start on a partially patched worktree	running
t+01:52	Suites exit `0`, no test imports the missing file	green
t+01:53	Verdict `pass`, artifact hash = full submitted diff	recorded
t+09:30	Fresh checkout fails to import the new symbol	detected

The detection came from a clean-tree integration run. It did not come from the gate.

Reproducing The Drill

The harness is deliberately small. It needs git, python3, and pytest.

git clone --depth 1 <your-repo> gate-drill cd gate-drill python3 -m venv .venv && . .venv/bin/activate pip install -q pytest # 1. Baseline: how many tests exist on a clean tree? pytest --collect-only -q | tail -n 1 # 2. Reproduce the bad path: per-file apply, warnings only python3 harness/split_apply.py --patch fixtures/five-file.diff --continue-on-error # 3. Observe a partially patched tree git status --porcelain

Step 3 is the diagnostic moment. A healthy gate never leaves you there.

Root Cause

The splitter is the origin of the defect. It converts one atomic diff into five independent patches.

# harness/split_apply.py (abridged, buggy on purpose) for chunk in split_by_file(patch_text): result = sh(['git', 'apply', '--3way', '-'], stdin=chunk, check=False) if result.returncode != 0: warnings.append(result.stderr.strip()[:200]) # <-- abort belongs here

git apply is all-or-nothing for the patch you hand it. Handing it five patches discards that guarantee.

Contributing Factors

Non-atomic apply. The loop traded one atomic operation for five independent ones.
Exit codes as logs. Failures were collected into a warnings list, not a control-flow branch.
No collection floor. The suite ran without comparing against a baseline count.
Wrong provenance field. The verdict stored the submitted diff hash, not the applied file set.

Each factor alone is survivable. Together they produce a green gate on a half-applied change.

The Durable Fix

1. Apply Once, Atomically

Pass the whole diff to git apply. Run --check first. Never loop per file.

2. Compare Declared And Actual File Sets

Parse the diff headers. After applying, parse git status --porcelain. If a declared path is absent, roll back and reject.

# harness/patchgate.py # Reference implementation. Adapt it. It is not a tested library. import hashlib, pathlib, re, subprocess DIFF_HEADER = re.compile(r'^diff --git a/(\S+) b/(\S+)$', re.MULTILINE) def sh(args, cwd, check=True, stdin=None): r = subprocess.run(args, cwd=cwd, input=stdin, capture_output=True, text=True) if check and r.returncode != 0: raise RuntimeError(f'{args[0]} failed: {r.stderr.strip()[:300]}') return r def declared_files(patch_text): return {m.group(2) for m in DIFF_HEADER.finditer(patch_text)} def changed_files(worktree): out = sh(['git', 'status', '--porcelain'], worktree).stdout return {line[3:].strip() for line in out.splitlines() if line.strip()} def tree_id(worktree): head = sh(['git', 'rev-parse', 'HEAD'], worktree).stdout.strip() status = sh(['git', 'status', '--porcelain=v1', '-z'], worktree).stdout return hashlib.sha256((head + status).encode()).hexdigest()[:16] def apply_all_or_nothing(worktree, patch_path): text = pathlib.Path(patch_path).read_text() want = declared_files(text) pre = tree_id(worktree) check = sh(['git', 'apply', '--check', '--3way', str(patch_path)], worktree, check=False) if check.returncode != 0: return {'verdict': 'reject', 'stage': 'precheck', 'detail': check.stderr.strip()[:400]} sh(['git', 'apply', '--3way', str(patch_path)], worktree) got = changed_files(worktree) if not want <= got: sh(['git', 'checkout', '--', '.'], worktree, check=False) sh(['git', 'clean', '-fdq'], worktree, check=False) return {'verdict': 'reject', 'stage': 'partial-apply', 'missing': sorted(want - got)} return {'verdict': 'applied', 'pre': pre, 'post': tree_id(worktree), 'files': sorted(want & got)}

3. Enforce A Collection Floor

Zero-test runs and shrinking suites look identical to passing suites at exit code level. Compare against a pinned baseline.

def test_floor(worktree, baseline): out = sh(['pytest', '--collect-only', '-q'], worktree).stdout counts = re.findall(r'(\d+) tests? collected', out) collected = int(counts[-1]) if counts else 0 if collected < baseline: return {'verdict': 'reject', 'stage': 'collection', 'collected': collected, 'baseline': baseline} return {'verdict': 'collected', 'count': collected}

Store the baseline next to the suite config. Refresh it only through review.

4. Record What Was Applied

Persist pre, post, and the sorted applied file list with every verdict. A reviewer can then diff the recorded set against the submitted diff without rerunning anything.

Verdict Decision Table

Stage	Condition	Verdict	Next step
Precheck	`git apply --check` non-zero	reject	re-request the diff at the current base
Apply	declared files are a subset of actual	applied	continue to collection
Apply	declared files are not a subset	reject, roll back	log missing paths, alert owner
Collection	collected below baseline	reject	treat as infrastructure error, not a test failure
Tests	exit `0` and collected at or above baseline	pass	persist pre/post tree ids
Tests	exit non-zero	fail	normal review path

Two reject stages are cheap. Both fire before any reviewer time is spent.

Where Free Model Access And A Free Server Fit

This drill needs many model runs and a disposable worktree. That is exactly where a free tier helps.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is described by its operator as offering free model access and a free server option. Two uses follow from that, both narrow. The first use is a drift sweep. Run the same five-file fixture against several available models. Count how often a model emits a diff that fails the declared-versus-actual check. The second use is hosting the harness on the free server, so repeated drills do not consume paid CI minutes.

The harness stays model-agnostic. Swapping a provider should change one argument, not the gate.

Quotas, the model roster, and server specifications change without notice. Verify current terms on the project page before quoting any number. Old benchmark posts are the wrong source for this.

Limitations

The harness checks application and collection, not semantic correctness.
A passing suite still proves very little about unseen call paths.
--3way can resolve hunks you did not intend to resolve.
Deleted-file handling needs extra care in the file-set comparison.
The code above is a reference implementation, not a maintained library.
Drift sweeps over free models are a sample, not a ranking.

Who Should Not Use This

Teams without a pinned baseline count should fix that first. The collection floor depends on it.

Repositories with intentionally variable suite sizes should use a ratio band instead. Anyone expecting a gate to approve semantic correctness should look at review, not automation. Anyone needing contractual uptime guarantees should not run production gates on a free tier.