A five-point lead between two agents on the same item set is frequently a property of the pairing, not of the models. Independent averages discard the single most useful fact in the run — that both arms answered the exact same questions — and that shared structure is the cheapest noise reducer available at small sample sizes. Pairing per item, seeding both arms identically, and reporting an interval on the per-item difference costs roughly thirty lines of code and one afternoon of setup.
Most comparison write-ups do the opposite. Each arm becomes a mean with a spread, the two means get subtracted, and the gap gets narrated as if it were measured. Item difficulty usually dominates model-to-model variation on real datasets, so discarding the item index inflates the standard error and produces an interval wide enough to swallow the headline. Pairing keeps the item difficulty term in both arms and cancels it in the subtraction, which is why the same sample size can separate two models that looked tied under independent aggregation.
The harness below is deliberately boring. It takes a pinned item list and a fixed seed list, calls both arms through the same OpenAI-compatible chat endpoint, scores each response with a scorer the operator supplies, and reduces everything to one paired delta. Nothing in it depends on a specific provider, and the only environment variables are a base URL, a key, and two model identifiers read from whatever dashboard the operator is using.
# paired_eval.py — stdlib only, no provider SDK import json, os, random, statistics, urllib.request def call(model, prompt, seed): # The path below is the common OpenAI-compatible shape. Confirm it against # the provider's own documentation before running anything expensive. body = { "model": model, "seed": seed, "temperature": 0, "messages": [{"role": "user", "content": prompt}], } req = urllib.request.Request( os.environ["MC_BASE_URL"].rstrip("/") + "/chat/completions", data=json.dumps(body).encode("utf-8"), headers={ "Authorization": "Bearer " + os.environ["MC_API_KEY"], "Content-Type": "application/json", }, ) with urllib.request.urlopen(req, timeout=90) as resp: return json.load(resp)["choices"][0]["message"]["content"] The scoring step stays outside the harness on purpose. A grader that lives inside the measurement loop is a grader nobody re-audits, and swapping it should never require touching the transport code. The only contract is that score(item, response) returns a float in a fixed range, and that the same scorer object is used for both arms.
SEEDS = [11, 12, 13] # shared across arms, never regenerated per model def run_arm(model, items, score): per_item = {} for item in items: runs = [score(item, call(model, item["prompt"], s)) for s in SEEDS] per_item[item["id"]] = statistics.fmean(runs) return per_item The reduction step is where the actual claim gets made, so it should be one function with no model names in it. A percentile bootstrap over the per-item differences is adequate here; it makes no normality assumption and it survives the many ties that binary scorers produce. Ten thousand resamples on a few hundred items finishes in well under a second.
def paired_delta(arm_a, arm_b, resamples=10_000, alpha=0.05, seed=7): ids = sorted(arm_a) assert ids == sorted(arm_b), "arms must cover identical item ids" diff = [arm_a[i] - arm_b[i] for i in ids] rng = random.Random(seed) n = len(diff) boots = sorted(statistics.fmean(rng.choices(diff, k=n)) for _ in range(resamples)) lo = boots[int((alpha / 2) * resamples)] hi = boots[int((1 - alpha / 2) * resamples) - 1] return statistics.fmean(diff), lo, hi, n Running it looks like this, with the model identifiers and base URL taken from the provider dashboard rather than hardcoded. The assertion on identical item ids is the guard that stops a partially failed run from silently becoming a comparison between two different datasets.
export MC_BASE_URL="<base url from the provider dashboard>" export MC_API_KEY="<your key>" export MODEL_A="<arm a identifier>" export MODEL_B="<arm b identifier>" python -m paired_eval --items items.jsonl --seeds 11,12,13 --out report.json Reading the result is a discipline question rather than a statistics question. If the interval straddles zero, the honest sentence is that no separation was detected at this sample size, together with the interval that bounds the effect. A point estimate quoted alone, even a correctly computed one, invites the reader to assume a precision the run never had.
# Format illustration only. Not measured results. arm_a mean 0.71 arm_b mean 0.68 n=240 items paired delta +0.03 95% CI [-0.01, +0.07] -> no separation detected unpaired delta +0.03 95% CI [-0.05, +0.11] -> same point, ~2x the width | Situation | Pair by item? | Why |
|---|---|---|
| Both arms answer the same prompts | Yes | Cancels item difficulty, narrows the interval |
| Arms see different prompt subsets | No, stratify instead | Pairing is undefined across disjoint items |
| Binary pass/fail scorer, few items | Yes, with repeats | Ties are common; pairing plus seeds stabilises the delta |
| Latency or cost comparison | Partly | Pair by item, but report per-item ratios, not means |
| Single arm, absolute score | Not applicable | Nothing to pair against; use the interval on the mean |
Free model access changes the economics of this workflow more than it changes the statistics. Running two arms over a few hundred items with three shared seeds is a few thousand short calls, which is exactly the shape of job that a free allotment can absorb without a procurement conversation. MonkeyCode's free model access and its free server option are relevant here for that reason: the harness is cheap per call but tedious to babysit, so hosting it on a free server means the run survives a closed laptop lid. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free allotment is time-sensitive by nature, so confirm the current limits on the provider's own page at signup rather than trusting any number quoted in a blog post, including this one.
The limitations are real and worth stating plainly. Pairing removes variance from item difficulty and nothing else, so it will not rescue a noisy scorer, an endpoint that drifts under load, or a dataset where the two arms were accidentally given different context. It also requires a frozen item list across the whole run, which is harder than it sounds once someone edits a prompt template mid-experiment. And with forty items, even a perfect paired design yields an interval too wide to separate anything short of a landslide.
Teams that need one headline number for a launch page should not use this method, because the output is often a refusal to rank. Groups comparing genuinely different task sets should stratify by task family first, since pairing across disjoint items is undefined. Anyone deciding purely on latency or price per call should skip the accuracy machinery entirely and measure ratios instead.
A forty-item pilot with three shared seeds is enough to see whether the pairing helps on a given dataset, and it is small enough to run twice before trusting the harness. Bring a frozen item list, a scorer that someone else has reviewed, and a willingness to publish the interval next to the delta.
