Five people just won $500 each for testing AI models properly. The Kaggle Benchmarking Challenge asked developers to stop reading someone else's leaderboard and build their own — and the five winning benchmarks, announced this week, have a pattern in them that every future benchmark should steal.
I read all five. Here is what each one actually tested.
Every model got Calgary wrong. Jonathansolvesstuff asked 19 of the newest models a simple question: if it is 9 a.m. in Calgary in November, what time is it in Toronto? Alberta stopped changing its clocks in June, so the answer is 10:00. Every single model said 11:00. The answer key comes from the public time zone database rather than anything written by hand, so the test can be run again every time that database updates. The models were not reasoning about time zones. They were reciting the rule they learned, which stopped being true in June.
The models read the changelog, then misapplied it. Nazar-boyko noticed that common advice about how much memory PHP arrays and Go goroutines use has quietly stopped being true. So he measured the real numbers himself across four versions of each language, then asked 12 models the same questions. The best models could quote the right release note but applied it to the wrong situation, and most reported high confidence even when they were wrong. Knowing the fact and knowing when it applies turned out to be two different skills, and the models have only one of them.
Running is not knowing. Rustam335 asked ten models to write Blender scripts for four specific versions of the software, then actually ran every script in those versions to see what worked. Plenty of models produced code that ran fine while stating nothing had changed between versions. Making each answer list the changes it relied on is what exposed the gap. The test is not "does the code run." The test is "does the model know why it runs."
Would today's AI have caught Ariane 5? Kaichen_dev rebuilt 26 of the most expensive software failures in history as small code samples, then asked 13 models to review them as if the code had not shipped yet. The models flagged a problem almost every time, but frequently for the wrong reason — including spotting SQL injection in code that never touches a database. The test also included disguised and already-fixed versions of each case, to catch models that merely recognise the famous bug and models that cry wolf on working code. Flagging something is easy. Flagging the right thing, and staying quiet when there is nothing to flag, is the actual skill, and it is the one the models keep failing.
Spell it wrong first, and a third of the answers keep it. Danielbutnar tested 43 models on two Romanian letters that look almost identical to their Turkish counterparts, where typing the wrong one is a common mistake. The models spell them correctly when the input is clean, but copy the wrong letter back about a third of the time when the input has it wrong. It even spread into words the models chose themselves, and the one-sentence fix for Romanian promptly broke Turkish, where the other letter is correct. The model did not know the letters. It knew the shape of the task.
Five different benchmarks, one shape. Every winner built a trap the model could not see coming: a law that changed in June, runtime numbers the author measured himself, code that had to actually run, famous bugs in disguise, a spelling trap that lives in the input. And every winner's answer key came from somewhere outside the author's own head — a public database, measured numbers, a real runtime, a fixed version, the other language's alphabet. That is the difference between a benchmark and an opinion. An opinion is a claim the author wrote. A benchmark is a claim the world can check.
The other thing the five share: none of them tested whether the model is smart in general. They tested whether it is right about one specific thing, in one specific situation, where being wrong has a visible cost. That is a much smaller question than "how good is this model," and it is a much more useful one. A leaderboard tells you which model wins on average. A benchmark like these tells you which model to trust with your clocks, your changelogs, your scripts, your code reviews, your spelling.
The challenge paid $500 to each winner, plus a DEV++ membership and a badge. Cheap, for what they produced. If you are building anything that depends on a model being right — and by now, that is most of us — the lesson is not "trust the leaderboard." It is "build the trap." Find the thing your model cannot have memorised, give it an answer key you did not write, and watch what happens. The winners just showed you five ways to do it.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam
