I Gave Four AIs the Same Spec. All Four Aced It.

I ran the fourth file through the test suite and watched 39 dots print out. Again.

....................................... [100%] 39 passed in 0.05s 

Same output as the third file. Same as the second. Same as the first. Four different AIs, one spec, one hidden suite — and not one of them dropped a single test. I had built the suite to bite. It bit the three broken versions I wrote to check it, exactly where they were broken. It did not bite a single AI.

The claim on the table

Every week brings a new leaderboard saying which AI writes the best code. The numbers come from benchmarks with names and versions, run by the labs or by people with strong opinions and a scoring script. I have never trusted a leaderboard I cannot re-run, so this series does the small, checkable version instead: one concrete job, one spec written in plain words, one test suite nobody showed the models, first drafts only. The tests decide. Everything else is colour.

The question for this episode: you hand four AI tools the same spec and nothing else — which one's first draft actually passes?

The lab

The job is a small module called flatbill. It splits shared-flat bills: an even split, a weighted split, and a late fee. All money is integer paisa, standard library only, three functions, no floats anywhere.

I picked this job because it is real work — the kind of thing a small app actually needs — and because it is full of the small decisions where code quietly goes wrong. Leftover paisa have to land somewhere exact: one each, to the first names alphabetically. The weighted split has a tie-break rule: largest fractional remainder wins, ties broken by name. A weight of 0 pays exactly 0 and never gets a leftover paisa. The late fee has a 3-day grace period, then 1% per day, rounded half-up (10.5 becomes 11, not 10), capped at 20% rounded down. And there is a leap-year test in there, because someone always forgets February.

None of these are hard. All of them are easy to get slightly wrong. That is the point.

The suite is 39 tests, written before any AI saw the spec, frozen at 21:41 on 12 November 2026 with its hash recorded (82a72d29…e553a47 — the full hash is in the data pack). The AIs never saw the tests. They got the spec and nothing else, the same words pasted verbatim, one fresh chat each, default model, one shot each. I did not paste failures back. I did not ask for fixes. None of them asked me a question, so the standard nudge I had ready — "use your best judgement and deliver the file" — never got used.

Before the runs, I checked the suite could actually fail things. I wrote three broken versions of the module on purpose: one that rounds naively and never fixes the remainders, one with no grace period, one that breaks ties by input order, rounds half-to-even and skips the cap. They scored 32, 33 and 34 out of 39 — each failing only its own bug's tests. The suite discriminates. It just never got the chance to discriminate between the AIs.

One more honest detail from before the freeze: the suite caught a mistake of mine. A remainder test used 1001 paisa over 3 people and expected one leftover paisa. Wrong — 1001 % 3 is 2, and the expected dict summed to 1000, not 1001. My reference implementation produced the correct answer and exposed it. I fixed the test before the freeze, and that was the last change the suite ever got.

What happened

The runs happened on Thursday 12 November, between 21:43 and 22:00, back to back in one sitting. The models themselves were fast. Copilot answered in eleven seconds.

ChatGPT, 21:43. Under a minute, one code block, no explanation, no questions. One thing I cannot tell you about this run: the page showed no model name anywhere. The responses were labelled only "ChatGPT". So the episode cannot name its model, and I will not guess.

Claude, 21:45. Under a minute, one code block, no explanation. Model shown: Sonnet 5.5 Medium.

Copilot, 21:51. Eleven seconds, prompt to complete file. New conversation in the Copilot chat web UI, Chat mode, no repository attached. One protocol wrinkle, recorded honestly: the instructions said "ask mode", but this UI has no separate Ask mode — the selector offers Interactive and Autopilot, so Interactive (the default) was used. Model shown: gpt-5.6-luna, 0.51 AI credits.

Gemini, 21:59. Under a minute, one code block, no explanation. Model shown only as "Gemini Auto".

Then I scored all four files against the frozen suite. 39, 39, 39, 39.

The verdict

Reference implementation, hand-written, for calibration: 39/39, 68 lines.

A four-way tie. I will say plainly what that means and what it does not. It means all four tools, given a well-specified small job with every edge case spelled out in the spec, produced a correct first draft on the same evening. It does not mean they are equal. It means this spec, at this size, with this suite, could not separate them. The suite was built to catch the fiddly bits, and all four got the fiddly bits right. If you wanted a ranking, this episode does not give you one — it gives you a tie and the receipts, and the honest next step is a harder spec.

What the numbers don't show

The files differ in ways the tests never see. Claude validates the most strictly: it checks that totals, weights and amounts are real integers (and not booleans), and it parses dates by hand — length, dashes, digit checks, then a calendar check. Copilot is the only one that checks the date format with a regex before parsing it, and its file is the shortest at 83 lines. Gemini is the only one that imports the datetime module instead of pulling date out of it, and the only one that put a docstring on all three functions; its cap is written as amount_paisa // 5, which is the same 20% rounded down, stated more plainly than anyone else's. ChatGPT's date check parses the date and then compares the round-trip — if the parsed date does not format back to the exact string you gave it, it refuses.

None of that changed a score. All of it is visible in the files, and the files are in the pack.

The limits, stated plainly

• One run per AI is a snapshot, not a ranking. These models are non-deterministic. The same prompt tomorrow might give a different file. What this episode can honestly say is what happened on this day, with this spec, on the first draft each tool produced.
• The tie is a finding about the spec as much as the tools. A harder spec — messier requirements, a bigger module, a bug to find rather than a function to write — is the obvious next test, and it is on the list.
• Model names are as shown, and two of them barely showed. ChatGPT showed no model name at all; Gemini showed only "Gemini Auto". "ChatGPT" and "Gemini" here mean whatever those pages served that evening.
• Small job, one evening. A ~90-line module, 39 tests, four tools, one sitting on 12 November 2026. No run was repeated, and no AI got a second attempt.

Run it yourself

The spec, the frozen suite and all four delivered files are in the data pack, with the scoring script. From the pack folder, python3 run_against.py runs/<ai>/flatbill.py scores any file against the same frozen suite — the reference implementation should come back 39/39, and the three broken variants should come back 32, 33 and 34, each failing only its own bug's tests. The chat links are in the pack too, so you can read each run where it happened.

If you run the same spec through your own tool of choice and the suite disagrees with my tie, that is data too. Tell me.

The full Episode 2 files — the spec, the frozen suite with its hash record, all four delivered files with their raw pytest outputs, the scoreboard and the method and protocol sheets — are in the members' data pack on Patreon, live now, under "LGTM S2 — The Four-Way Tie: Complete Files". The articles stay free; the receipts are there if you want to audit mine.

Next episode, Tuesday 19:00: a real open-source issue, a real Copilot fix attempt, and the repo's own test suite as the judge. No more green-field specs — this time the code already exists, the bug already bites, and the tests were written by someone else.

So, the question I'll leave open until then: if four different AIs all ace the same well-written spec on the first try, is the skill still in the writing of the code — or has it moved, quietly, into the writing of the spec?

P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam