Asking the judge to solve the problem itself before grading turned a rubber stamp into a real check.
I built a small local judge to grade a weaker model's math homework. I fed it 41 answers I had deliberately made wrong, expecting it to catch most of them. It caught one. The other 40 wrong answers, some off by a single cookie from a rounding slip and some simply fabricated, sailed through marked CORRECT, because the judge never once reached for a calculator. It read the student's confident-sounding paragraph, saw a clean "FINAL ANSWER:" line at the bottom, and signed off. Then I changed one thing about the prompt: instead of asking the judge to check the answer, I asked it to solve the problem itself first and only then compare. Same model, same 41 answers, same everything else. It caught all 41.
I wrote this with AI assistance, and I fact-checked and edited every claim myself.
Nothing about this started out as a story about rubber-stamping. It started as an attempt to measure something more fashionable: whether a judge model goes easy on text that reads like its own output, or whether longer, more hedging-heavy answers talk their way past a grader regardless of whether the number at the end is right. Neither of those turned out to be the real story in my own data, and I'd rather report the finding that actually showed up than force the one I went looking for.
The experiment, in one card
What was actually run: two local Ollama models, llama3.2:1b (weak) and llama3.1:8b (strong), solving word problems, with llama3.1:8b also acting as judge in every condition. First task set: 20 easy two-step arithmetic problems, both models scored 20/20, a ceiling effect that left nothing to judge. Second task set: 20 harder three-step bakery/percentage problems with a floor-rounding step, where the weak model dropped to 9/20 correct, finally giving the judge 11 real wrong answers to catch. A separate 30-item design held one wrong number constant across three verbosity levels (terse, medium, hedging-heavy verbose) to test for a style tilt. Combined: 41 deliberately wrong answers, graded twice by the same judge model under two different prompts. Everything deterministic (temperature=0, seed=7). Full raw data, both prompts, and the chart below are pinned at commit f228d26.
A real mid-run problem, not hidden: the first task set was too easy for its own purpose. Both models solved every single item, so there was no wrong answer for the judge to be lenient or harsh about. I rebuilt a harder task set specifically to create a real accuracy gap, logged in methodology.md as a design iteration rather than quietly swapped out.
TL;DR
What the 41 judged answers actually showed:
- A plain one-shot judge prompt ("is this correct or incorrect?") caught 1 of 41 deliberately wrong answers, about 2.4%.
- The exact same judge model, given a chain-of-thought prompt that made it solve the problem itself before comparing, caught 41 of 41, 100%, on the same byte-identical candidate texts.
- The plain judge's one catch was the most verbose wrong answer in the set, which cuts against a simple "verbosity makes judges lenient" story rather than supporting it.
- The weak model (
llama3.2:1b) scored 9/20 on the harder task set; the strong model (llama3.1:8b) scored 20/20 on the same tasks, confirming the harder set actually produced wrong answers worth judging, unlike the first attempt. - The fix cost nothing extra in model calls, only a different prompt on the same judge, and generalized cleanly across two independently-run task designs.
Wrong-answer catch rate, same judge model, same 41 answers, one prompt change. Image by author.
The question this was built to test
Going in, the question was narrower and more fashionable than "does the judge check the arithmetic." The LLM-as-judge literature documents two specific tilts: self-preference bias, where a judge favors text that reads like its own output (arXiv 2410.21819 ties this to the judge's own perplexity on the candidate text), and verbosity bias, where longer or more hedging-heavy answers get rated more favorably at the same underlying correctness. Both are testable, and I set out to measure both with the same judge model grading its own and a smaller model's answers, blind to which model produced each one. That was the plan. The first task set couldn't answer it, and the second task set answered a more basic and more consequential question instead.
Method
Method, in enough detail to rerun it. gen_tasks_hard.py builds 20 three-step bakery word problems (total stock, minus full trays sold, minus a percentage used for a sample, rounded down), seeded with random.seed(11) so the exact numbers are reproducible. Both llama3.2:1b and llama3.1:8b solve all 20 through a prompt that asks for brief reasoning and a FINAL ANSWER: <number> line. The plain judge then sees only the problem text and the candidate's full solution, blind to which model wrote it, and is asked to reply CORRECT or INCORRECT on the first line. The chain-of-thought judge gets a different prompt: solve the problem yourself first, state your own number as MY ANSWER: <number>, then compare to the student's answer and give a verdict on the last line. Both judge prompts ran against the exact same 41 candidate texts; no new candidate generation happened between the two judging passes.
def cot_judge_prompt(task_prompt, candidate_answer_text): return ( "You are grading a student's solution to a word problem. First solve the problem " "yourself from scratch, showing your own work, and state your own final number on a " "line that starts with exactly: MY ANSWER: <number>. Then compare your answer to the " "student's final answer below. On the very last line, state your verdict as exactly: " "VERDICT: CORRECT or VERDICT: INCORRECT.\n\n" f"PROBLEM:\n{task_prompt}\n\n" f"STUDENT SOLUTION:\n{candidate_answer_text}\n" ) Results
Results. Plain judge: 1 of 41 wrong answers caught (2.4%). Chain-of-thought judge: 41 of 41 (100%). Broken down by where the wrong answers came from: in the 11 hard-task answers (a genuine self-preference-style design, both models solving blind), the plain judge caught 0, the CoT judge caught 11. In the 30 verbosity-held-constant answers (one wrong number, three presentation styles), the plain judge caught 1, the CoT judge caught 30. The CoT judge also kept all 29 genuinely correct answers in the hard-task set marked correct, zero new false positives there, so the fix isn't simply "mark everything incorrect and get credit for the wrong answers by accident." It is catching real errors without breaking on real correct ones, at least on this task set.
Where it actually broke
Where it actually broke. The clearest single case: a bakery problem where the correct answer was 131, and the weak model computed 15% of 153 as 23.05, rounded it up to 23 instead of down to the correct floor value 22, and landed on 130, off by exactly one cookie. The reasoning in its answer reads cleanly, step by step, with a confident FINAL ANSWER: 130 at the end. The plain judge's verdict: CORRECT, one word, no recomputation. The chain-of-thought judge, given the identical text, worked the same problem itself, landed on 131, compared it to the candidate's 130, and correctly flagged it. Full transcript in failure_examples/htask_01_plain_judge_rubber_stamp.md.
The one exception matters too. Across the 30 verbosity-held-constant answers, the plain judge caught exactly one, and it was the longest, most hedging-heavy variant in the set, the one that opens with "let me work through this carefully, step by step, to be sure" and closes with "cross-checking this arithmetic once more for safety." That is the opposite of what a simple verbosity-bias story predicts. With one data point, it isn't evidence that verbosity helps catch errors; it's a sign that the rubber-stamping failure was so dominant and so close to total that it swallowed whatever smaller verbosity or self-preference signal might have existed underneath it. I went looking for a subtle bias and found a much blunter problem instead.
What this means if you're using an LLM as a grader
What this means if you're using an LLM as a grader. If your judge prompt looks like "here's a problem and an answer, say correct or incorrect," and the task has a checkable right answer, don't trust its calibration against anything beyond the most obviously wrong text. This judge wasn't being subtly fooled by style; it mostly wasn't checking at all. The fix that worked here costs nothing extra in API calls: make the judge produce its own answer first, in its own context, before it ever compares. That forces the actual computation to happen somewhere, instead of asking the model to spot someone else's arithmetic error without redoing the arithmetic. It will not be this clean on tasks without a single checkable number, like code review or open-ended writing quality, where "solve it yourself first" is a much harder ask for the judge. But for anything with a programmatically or numerically checkable answer sitting inside an LLM-graded pipeline, this is a one-line, zero-cost change worth testing before trusting the grader's calibration at all.
My honest take: I didn't expect a near-total miss. I expected a judge that was somewhat lenient toward familiar-sounding text, not a judge that was barely grading at all. That gap between "subtly biased" and "not really checking" is, I think, the more useful thing to take from this for anyone wiring an LLM judge into a real pipeline: test whether it's checking before you worry about whether it's checking fairly.
Reproducibility
Reproducibility. Every task, both prompts, all 41 raw judged transcripts, the two annotated failure cases, and the chart script are public at github.com/priyansh18/medium-articles, in the llm-judge-rubber-stamp/ subfolder, pinned to commit f228d26. methodology.md covers the full design, including the ceiling-effect task set that didn't work and why; analysis.md has the complete numeric breakdown, including the stated limitation that the chain-of-thought judge's false-positive rate was only verified on the hard-task set, not the full 61 items across both designs.
If you're grading model outputs with another model right now: would you rather test your judge's calibration with a chain-of-thought rewrite like this one, or with a held-out set of answers you already know are wrong, the way this experiment did, before trusting its verdicts on the answers you don't already know?
If you want a deeper, structured path through this material rather than one article at a time, I've been building a free roadmap at notes.priyanshsinghal.com as I study it myself, and I take on 1:1 sessions for people working through the same material at topmate.io/priyansh_singhal10.
