If you run a coding agent for long enough, you end up maintaining two things: the model choice, and everything around it. The system prompt, the CLAUDE.md, the hooks, the tool wrappers, how much of a big file gets pasted back into context, when the loop may stop. Most of my time on the second pile is reading a failed session, guessing what went wrong, and adding a line somewhere.
A new paper argues that this pile is where most of the reliability lives, and that the guessing can be automated without the agent wrecking itself. It is Harnesses That Improve Themselves: A Unifying Framework for Self-Improving Agent Harnesses by Deep Shah and Nehal Kathrotia, both listed with Google, submitted on 17 November 2026. It is mostly a framework paper: it maps about fifteen recent systems that rewrite their own runtime code onto one architecture, then runs a small SWE-bench Lite experiment to test the design rules it pulls out.
These are my notes on how it works, a toy version of the loop you can run, and what I would change in a Claude Code style setup because of it.
The problem: a harness tuned for one model breaks on the next
The paper calls the runtime around the model the agent harness, and gives three reasons it matters as much as the weights. When the same model runs inside different harnesses, its main failure modes (quitting before testing, looping on failed commands, overflowing context, skipping verification) follow the harness, not the model family, and tokens per solved task can vary by an order of magnitude. A good harness props up weak models on short tasks and gives strong models the state tracking they need on long ones. And no single harness suits every model: one tuned for a model can flip leaderboard rankings with another, and hand-maintained harness releases often stall or regress older tasks.
That last point is the one I recognise. You fix the agent for today's model, the model changes, and half your fixes become dead weight or start hurting.
Earlier automated work tuned prompts, demonstrations or multi-agent call graphs but treated the inner runtime (compactors, tool wrappers, loop controllers, stop checks) as fixed. Self-improving harnesses treat that runtime as code the system may edit.
What the paper proposes: four editable parts and a four-stage loop
The framework has two halves. The inner loop is one task: a frozen model talks to a live environment through four code-editable components. The outer loop runs across many tasks and rewrites those components, in four stages.
The outer loop. Stage 3 is the gate that decides what ships. Simplified from the paper's Fig. 1.
The four components, in the paper's words reduced to what they do:
- Context and State: assembles instructions, separates working memory from longer-lived experiential memory, and masks or compacts stale observations so long runs do not overflow.
- Agent Control: turn budgets, retries, detecting and breaking loops, switching between planning and execution, handing work to subagents.
- Action Interface: the agent-computer interface. It turns model output into sandboxed shell commands, file edits or GUI actions, and truncates or formats what comes back.
- Verification and Recovery: intercepts the stop signal and runs tests, linters or visual checks, then routes failures back into context for repair instead of letting the run end.
Section II also writes the goal down as a constrained optimisation, which is the cleanest part of the paper. The harness is a four-tuple of code modules, the model is frozen, and the objective is the expected verifier pass rate, subject to a token budget and a validation gate. Here it is as a data shape:
eq1.txt
h = (h_ctx, h_ctrl, h_act, h_ver) # the harness: four code modules M_theta # the model, weights frozen R_x(tau) in {0, 1} # external, read-only verifier for task x C(tau) # runtime tokens of trajectory tau maximise E[ R_x(tau) ] over harnesses h subject to E[ C(tau) ] <= B_tok # per-task token budget G_val(h, h0) = 1 # Stage 3 gate vs the starting harness h0: # modified code paths actually ran, and # no regression on a held-out val split The important part is subject to. Unconstrained maximisation over a small dev set is how you get a harness that memorises the dev set. The modular version approximates this one module at a time, then runs a merge pass to resolve conflicts.
How the outer loop works, stage by stage
- Stage 1: Diagnose failures
Run the harness on a dev split, keep full trajectories, and find recurring bottlenecks by clustering verifier errors or diffing passing and failing runs. One failed run is not evidence.
- Stage 2: Propose edits
An optimiser (the task model itself, or a stronger coding model) turns a failure cluster into a code or hook patch.
- Stage 3: Validate and filter
Screen the candidate on validation tasks. Reject anything that regresses, burns tokens, or wins only on dev-set quirks.
- Stage 4: Deploy or route
Merge the patch, archive it for later rounds to branch from, or route each task to a harness variant.
The paper names three design choices that separate the surveyed systems. Who proposes the edit: the task model itself, or a separate stronger coder (needed for small or domain-specific agents). What code can change: just the task harness, or the optimiser too, as in the recursive Darwin and Mendel Gödel Machines, which costs more compute and needs strict sandboxing. Where the evaluator lives: always outside the agent's write scope, so it cannot pass by editing the grader or the logs.
Underneath: telling a harness bug from a model limit
Section III is the part I found most useful, because it answers the question I get wrong most often when reading failed sessions: was that the model, or my setup?
The paper's rule is a divergence check between what the model reasoned and what actually happened. If the reasoning trace names the right file or the right fix (or the task passes on at least one retry), but the run still fails because the edit tool rejected the patch format, the shell lost its working directory, compaction dropped an earlier constraint, or the loop exited before running tests, then the harness is at fault. If the tool calls ran cleanly, observations stayed intact, and every retry still fails on the domain logic, Stage 1 abstains: no harness patch, because spending search compute on a model knowledge gap is waste.
It lists four ways the surveyed systems attribute failures:
- Failure clustering: group failed runs by their compiler or test error, which filters out one-off slips.
- Contrastive diffing: diff a passing and a failing run of the same task.
- Backward causal tracing: walk back from the stack trace to the earliest turn that wrote or misread the broken value.
- Self-reports: the agent says where tools held it back, and a verifier keeps only claims backed by trace events.
This matters because of an ablation the paper cites from the Meta-Harness work: an optimiser that saw only scalar scores or short summaries stalled near baseline, while one with a searchable folder of raw trajectories, prior harness code and diffs found recurring bottlenecks. Reflecting on single failures is also how instruction files fill up with rules for problems that never repeat. One cited paper is titled "Why Does CLAUDE.md Keep Growing?"
On the patch itself, the paper summarises rule ablations from other work: explicit negative boundaries ("never exit without running the tests") beat step-by-step procedural advice, and plain-text safety rules are not enforced unless a deterministic hook blocks the tool call at runtime. Small lifecycle hooks (pre-turn, post-tool, pre-stop) are how the surveyed systems keep self-edits from touching the core loop.
What these loops keep inventing
Across coding, terminal, web, GUI and robotics benchmarks, the optimisers keep finding the same five families of fixes:
- Pre-stop verification: check the diff and run tests before the agent may exit. The paper calls blocking unverified exits one of the largest single jumps in solve rate.
- Empty-turn recovery: catch a turn with no tool call or a malformed action and inject a continuation prompt.
- Loop breakers: counters on repeated failing commands that force a change of strategy.
- Environment and memory persistence: carry the working directory and PATH across stateless shell calls, and keep working memory apart from reusable experience.
- State-commit guards: confirm a click changed the screen, or a robot's state is consistent, before moving on.
If you have built agents by hand, all five look familiar. They are what people add after reading enough failed runs.
Why full rewrites overfit, and the four safeguards
Section IV argues the unit of change should be one module, not the whole harness. Observation filtering, compaction, turn limits and completion checks share assumptions, so changing them together breaks credit assignment and lets one edit quietly break another (a new output filter trims the stack trace a new verification check needs). The paper cites a ModularRSI ablation where editing all modules at once did worse than the untouched seed on held-out terminal tasks, and an RRSI result where unrestricted evolution topped its dev split but gained almost nothing out of distribution while more than doubling tokens per task.
The fix is four safeguards, spread across the loop:
- Evolve modules separately, then run a merge pass that reconciles runtime conflicts using failure traces.
- Keep edits small and general: each patch traces to a recurring failure cluster, edit budgets shrink over generations, task-specific hints are rejected.
- Enforce acceptance checks: no regression on evolution and validation splits, and proof the new code path actually executed. Code that never runs can pass by luck.
- Separate dev and test pools completely, and freeze the harness before out-of-distribution testing.
Section V adds a finding I did not expect, from a cross-scale study it cites: once a model clears a basic instruction-following bar (the example is 32B+ models), the patches it proposes are about as good as a frontier model's, because diagnosing traces is easier than solving the task. Structural checks also transfer across models far better than model-specific prompt wording.
The experiment: 142 lines of Python, one module at a time
Section VII uses 60 SWE-bench Lite tasks: 40 from sympy split into dev (12), validation (8) and held-out test (20), plus 20 from django as a held-out cross-repository test. Each run is in a clean sandbox with git history stripped, graded externally. Every method starts from the same 142-line seed harness:
seed harness
context_mgr.py prompt + raw turn history (no compaction) control.py 12-turn limit + JSON parser action_aci.py bash, view_file, edit_file, finish verify.py pass-through stop hook (lets every finish through) The task agent is Gemini 3.8 Flash, frozen. The optimiser is Gemini 4 Carbon (falling back to Gemini 4 Argon on timeouts). Five methods are compared: the seed once, best of three seed runs, prompt-only reflection that writes learned rules into a RULES.md, an unconstrained rewrite of all four files with no validation or token cap, and modular gated evolution: one module per step, Stage 3 validation, checks that new lines executed, and a 110k-token runtime cap.
The four proposed module edits. The control.py change improved dev accuracy and still got rejected. Source: Section VII.
The rejected edit is the interesting one. Raising the turn limit from 12 to 18 made dev accuracy jump, the kind of change you would happily ship after a good afternoon. Validation dropped and tokens broke the cap, so the gate threw it out. The unconstrained rewrite made the same kind of move (it raised the cap to 28) and nothing stopped it.
Held-out results across both repositories, 40 tasks. Best of three is left out: 5.0% at 237.6k tokens. Numbers from Table IV, Part A.
The numbers that stood out to me:
- Retries do not fix a harness bug. Best of three more than doubled tokens (237.6k per task) and stayed at 5.0%, because context overflowed the same way on every retry.
- A rule in a text file did not stop the behaviour it banned. The first rule in RULES.md forbade finishing with an empty or untested diff. The model still exited early: unverified cross-repo exits went from 15% to 30%, and accuracy fell to 2.5%. The same idea as a Python check in verify.py took unverified exits to 0.
- The full rewrite scored highest (42.5%) at 451.6k tokens per task, with one rollout over 2.8M tokens.
- Modular gated got 40.0% at 46.7k tokens per task: 8x the seed's accuracy at nearly 10x fewer tokens than the rewrite (116.8k vs 1,062.6k per solved task). The paper says the savings pay back its 100 search rollouts in under 100 deployment tasks.
- Transfer to a stronger model, with no re-tuning: on Gemini 4 Argon, the modular harness got 37.5% at 64.8k tokens, the full rewrite fell to 30.0% at 284.7k, prompt-only got 15.0%, and the seed 10.0%.
Warning: Read these numbers at the size they are. Each test split is 20 tasks, so one task is 5 percentage points, and it is one benchmark with one model family. The full rewrite actually beat the gated harness on the django split (50% vs 40%) and only lost on the in-domain split and on cost. The paper mostly argues from its survey of other systems; Section VII is a small check of those rules, not proof on its own.
A toy version you can run
To make the loop concrete I wrote a small simulation. This is not the paper's code, and its numbers mean nothing outside the toy. The fake model has fixed habits: it quits early a quarter of the time, re-reads files once context passes 4,000 tokens, and about one task in five is beyond it (a model limit). The harness is four knobs, one per component. The outer loop clusters failures, proposes one module patch from a playbook (standing in for an LLM optimiser), gates it on a validation split with a token cap and a did-the-code-run check, and deploys it if it passes.
harness_loop.py
# harness_loop.py: a toy version of the diagnose -> propose -> validate -> deploy loop. # Standard library only. The "model" is a fake with fixed failure habits, so it runs offline. import random from collections import Counter from dataclasses import dataclass, replace # ---- the harness: four editable modules, as plain config + flags ----------------- @dataclass(frozen=True) class Harness: ctx_keep_turns: int = 99 # Context & State: how many past observations stay raw ctrl_max_turns: int = 12 # Agent Control: turn budget act_view_window: int = 0 # Action Interface: 0 = view_file returns the whole file ver_block_unverified: bool = False # Verification & Recovery: no finish without diff + test @dataclass class Task: id: int file_tokens: int # size of the file the agent has to read edits_needed: int model_can_solve: bool # False = a model knowledge gap, not a harness bug def make_tasks(seed, n): r = random.Random(seed) return [Task(seed * 100 + i, r.choice([1200, 2400, 4200]), r.choice([1, 2, 3]), r.random() > 0.2) for i in range(n)] # ---- the frozen "model": deterministic habits, no API ------------------------------ CONFUSION_TOKENS = 4000 # past this much context, the fake model starts re-reading files QUIT_EARLY = 0.25 # chance per turn that it calls finish before it is done def run(h: Harness, task: Task): """Inner loop. Returns (solved, tokens, trajectory signature list, hooks that fired).""" r = random.Random(task.id) # same randomness for every harness version: a paired test history, tokens, events, fired = [], 0, [], set() read, edits, tested = False, 0, False for turn in range(h.ctrl_max_turns): if turn >= 12: fired.add("ctrl") # a turn the seed's 12-turn cap would not allow # Context & State: older observations get compacted to a 60-token summary ctx = sum(t if i >= len(history) - h.ctx_keep_turns else 60 for i, t in enumerate(history)) if h.ctx_keep_turns < 99 and len(history) > h.ctx_keep_turns: fired.add("ctx") tokens += 800 + ctx + 120 # system prompt + context + model output if r.random() < QUIT_EARLY and turn >= 2 and not (edits and tested): if h.ver_block_unverified: fired.add("ver"); history.append(80); events.append("finish_blocked"); continue events.append("finish_unverified"); return False, tokens, events, fired if ctx > CONFUSION_TOKENS and r.random() < 0.6: action = "reread" elif not read: action = "view" elif edits < task.edits_needed: action = "edit" elif not tested: action = "test" else: action = "finish" if action in ("view", "reread"): obs = h.act_view_window or task.file_tokens if h.act_view_window: fired.add("act") history.append(obs); read = True; events.append(action) elif action == "edit": edits += 1; history.append(150); events.append("edit") elif action == "test": tested = True; history.append(400) events.append("test_pass" if task.model_can_solve else "test_fail") if not task.model_can_solve: tested = False # it keeps trying edits that cannot work edits = 0 else: events.append("finish") return task.model_can_solve, tokens, events, fired events.append("turn_cap") return False, tokens, events, fired def evaluate(h, tasks): res = [run(h, t) for t in tasks] solved = sum(s for s, *_ in res) avg_tok = sum(tok for _, tok, *_ in res) / len(tasks) fired = set().union(*(f for *_, f in res)) return solved / len(tasks), avg_tok, res, fired # ---- Stage 1: diagnose failures by signature, abstain on model limits ------------- def diagnose(tasks, results): clusters = Counter() # one failed run can carry several signatures for task, (solved, _, ev, _) in zip(tasks, results): if solved: continue if ev.count("reread") >= 2: clusters["context_overflow"] += 1 if ev[-1] == "finish_unverified": clusters["unverified_exit"] += 1 if ev[-1] == "turn_cap" and not {"reread", "test_fail"} & set(ev): clusters["turn_cap"] += 1 if "test_fail" in ev and "reread" not in ev: clusters["model_limit"] += 1 # tools were fine; do not patch the harness return clusters # ---- Stage 2: propose ONE module patch per round (a stub optimizer) --------------- PLAYBOOK = { # signature -> candidate patches, in the order an optimizer might try them "context_overflow": [("control: raise turn cap 12 -> 18", "ctrl", {"ctrl_max_turns": 18}), ("context: compact obs older than 2 turns", "ctx", {"ctx_keep_turns": 2}), ("action: windowed view_file (400 tok)", "act", {"act_view_window": 400})], "unverified_exit": [("verify: block finish without diff + test", "ver", {"ver_block_unverified": True})], "turn_cap": [("control: raise turn cap 12 -> 18", "ctrl", {"ctrl_max_turns": 18})], } # ---- Stage 3: the gate -------------------------------------------------------------- B_TOK = 40_000 # per-task runtime token budget def gate(base, cand, cand_fired, hook): if hook not in cand_fired: return False, "new code never ran on val" if cand["val"] < base["val"]: return False, "regressed on val" if cand["tok"] > B_TOK: return False, f"avg tokens {cand['tok']:,.0f} > cap" if cand["dev"] <= base["dev"] and cand["val"] <= base["val"]: return False, "no gain" return True, "accepted" def main(): dev, val, test = make_tasks(1, 20), make_tasks(2, 20), make_tasks(3, 40) h, tried = Harness(), set() print(f"seed harness: {h}") for rnd in range(1, 7): d_score, d_tok, d_res, _ = evaluate(h, dev) v_score, v_tok, _, _ = evaluate(h, val) base = {"dev": d_score, "val": v_score, "tok": v_tok} clusters = diagnose(dev, d_res) print(f"\nround {rnd}: dev {d_score:.0%} val {v_score:.0%} tok/task {v_tok:,.0f}") print(" clusters:", dict(clusters.most_common())) target = next(((sig, p) for sig, _ in clusters.most_common() if sig != "model_limit" for p in PLAYBOOK.get(sig, []) if (h, p[0]) not in tried), None) if not target: print(" no harness patch left to try: stop"); break sig, (name, hook, change) = target tried.add((h, name)) # a rejected patch may be retried once the harness changes cand_h = replace(h, **change) cd, _, _, _ = evaluate(cand_h, dev) cv, ct, _Source:https://dev.to/ahmad_dogar_7d098f5e48f72/harnesses-that-improve-themselves-how-an-agent-can-rewrite-its-own-runtime-without-breaking-it-1p2iRelated posts



