Bisect Prompt Blocks to Localize a Silent Regression

When a suite score slides from 0.86 to 0.79 and nobody touched the model, the useful question is which prompt block caused it. Most silent regressions tend to sit inside exactly one edited block rather than spread evenly across the whole string. So run a block bisection before you re-run averages, swap models, or start rewriting cases you may not need to touch.

Bisection is cheap, boring, and falsifiable, which is precisely why it survives contact with a deadline. It takes the candidate prompt, reverts one labeled block to its previous version, and scores the suite again under paired repeats. If the score recovers, that block owns the loss; if it does not, you have eliminated a suspect in a single sweep.

Make blocks explicit before you measure anything

You cannot bisect a prompt that arrives as one undifferentiated string. Split the prompt into named blocks with delimiters at assembly time, keep each version as a YAML file, and let git carry the diff between them. The delimiters matter twice: they make the boundaries visible to the model and they make the report readable by a human at 2 a.m.

# prompts/support_v3.yaml version: 3 blocks: role: | You are a support triage assistant. Classify each ticket and cite the policy clause. rules: | If the ticket asks for a refund past 30 days, route to billing_review. Never invent an order id. Ask for it when it is missing. format: | Respond with JSON: {"route": str, "confidence": 0..1, "citations": [str]} 

Assembly is a join, and the join has to stay stable across baseline and candidate or the comparison means nothing. This helper is the only place where block order is decided, so a reordering bug cannot sneak into a probe.

def assemble(blocks: dict[str, str]) -> str: return "\n\n".join(f"<{name}>\n{text.strip()}\n</{name}>" for name, text in blocks.items()) 

The probe loop

The probe takes the candidate blocks, swaps exactly one block back to the baseline value, and scores that hybrid prompt. Everything else stays fixed, so the only variable in the measurement is the block you named in the loop.

import statistics def score_prompt(provider, prompt: str, cases, repeats: int = 3) -> float: runs = [] for _ in range(repeats): graded = [grade(case, provider.complete(prompt, case.input)) for case in cases] runs.append(sum(g.credit for g in graded) / len(graded)) return statistics.fmean(runs) def probe(block: str, base: dict, cand: dict, provider, cases, repeats: int = 3) -> dict: hybrid = dict(cand) hybrid[block] = base[block] hybrid_score = score_prompt(provider, assemble(hybrid), cases, repeats) return {"block": block, "hybrid": hybrid_score} 

Paired per-case deltas decide the verdict, not the mean on its own. Compare the baseline prompt and the hybrid prompt on the same cases in the same order, then count wins and losses per case. A block that flips eleven cases up and two down is a real suspect, while a block that nudges the mean by 0.01 through repeat noise is not.

python bisect_blocks.py \ --baseline prompts/support_v2.yaml \ --candidate prompts/support_v3.yaml \ --cases evals/support.jsonl \ --repeats 3 \ --out reports/bisect-2026-11-07.json 

The sweep is linear in the number of blocks, which is the honest price of ablation without interaction modelling. Six blocks times forty cases times three repeats is 720 graded calls for one full sweep, and that arithmetic sets your budget before you start. When those calls are metered, people quietly skip the sweep and guess instead, which is how a fixable regression becomes a folklore theory about model quality.

Where free model access changes the economics

Ablation is exactly the workload that dies on a per-token invoice, because every probe is a full suite pass that produces no new features at the end. MonkeyCode's project advertises free model access and a free server option, so a sweep of this shape can run in a free cloud workspace while your laptop stays on battery. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance is operator-supplied and promotional terms change over time, so read the current limits on the project page before you size a sweep against them. Nothing in this workflow depends on the price staying at zero, because the price only decides how often you run the sweep, not whether the method works.

Reading the report without re-deriving the logic

The report is a small JSON document with one row per block, plus the paired win and loss counts that justify each row. Keep the interpretation table in the repository next to the script, so the next person inherits a decision rule rather than a mystery file.

Two blocks that each recover nothing but together recover everything are an interaction, and the honest conclusion is that your block boundary is wrong. That is still a result worth recording, because it tells you the rules and the format must be edited together from now on. Write that sentence into the prompt's changelog while the evidence is fresh in the report.

Limitations and who should skip this

Bisection localizes a regression; it does not explain one, and you still have to read the diff and form a hypothesis about why the wording mattered. It assumes your cases discriminate, so a suite where every case passes or every case fails yields a tidy report with no information inside it. Grader noise puts a floor under the smallest delta you can trust, and a model-based grader needs its own repeat policy before it is allowed to judge anything else.

Skip this if your prompt has no stable block structure, if your suite is under roughly twenty cases, or if your team does not version prompts at all. Fix the versioning first, because a bisection between two uncommitted strings is an anecdote wearing a chart, not a measurement. Teams shipping a single-block prompt should invest in case quality before they invest in localization tooling.

Run the sweep once on an unchanged prompt before your next release and keep that report as a control. When the next score drop arrives, you will already hold a cost estimate, a decision table, and one named block instead of a week of guessing. If you want to try the loop without watching a meter, open a free MonkeyCode workspace, paste the script above, and let the first sweep finish while you read its output.