Originally published in Generative AI on Medium.
I spent months building a system that wraps a language model in a checking loop so it can prove hard math theorems. Every step gets verified with real math tools, the system remembers what worked, and a second pass tries to break the proof before it counts.
Then I tested it on 300 hard math problems, and the results surprised me: the setup around the model matters as much as the model itself.
What it does
The harness is three things:
- A checking loop. Every intermediate claim is verified in code (SymPy, Z3, 50-digit mpmath). The solver works on a persistent scratchpad; a separate auditor runs in a fresh kernel and tries to kill the proof.
- Memory. After each problem, a reflection pass distills what worked into heuristics the system reuses, with zero weight fine-tuning.
- Governed search. No proof drafting on turn 1, context collapses to a clean blackboard after rejections, and an overtime gate rescues near-misses.
The results
| Model | One pass | With the harness | Multiplier | Failures rescued |
|---|---|---|---|---|
| Qwen-3.8-27B (local, one RTX 5090) | 12.67% | 26.67% | 2.11x | 16% |
| Meta Muse 1.3 | 22.00% | 46.33% | 2.11x | 31% |
| Gemini 3.8 Flash | 33.67% | 55.33% | 1.64x | 33% |
| 3-model portfolio | 41.33% | 65.67% | 1.59x | 41% |
The honest headline is the last column: the loop fixed 16–33% of the problems each model got wrong on its own. Flash is the best model and improves least in multiples (1.64x, not 2x) but most in failures rescued.
The detail I'm proudest of: the local 27B catches factor-of-2 and boundary-case errors both big models miss. Five problems only Qwen solved; it beat Muse on 14 problems and Flash on 10. Flash is also 7.8x faster than Muse at the median solve (12.2 vs 95.0 minutes).
The honest limitations
Proofs are graded by our own calibrated LLM judge ensemble, running in an isolated context — no human grading, no formal verification. All the eval logs are public so anyone can recount: certified counts reproduce exactly from the judgment files (Qwen 80, Muse 139, Flash 166).
Read the full story
The full article is published in Generative AI on Medium: https://generativeai.pub/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-231314ab3ea7
The code and all test results are open: https://github.com/dionyziuz/proof_harness — eval logs: https://github.com/dionyziuz/proof_harness/tree/main/data/eval_results
If you try it or spot a flaw in the method, I'd like to hear it — the comments here or an issue on the repo.

