Mining 14,351 Chess Puzzles with Stockfish 18 (and the Illegal-FEN Trap in chess.js)

When we started building Brass Pawn (source on GitHub, AGPLv3), we needed a large offline tactics, positional judgement, and endgame dataset that ran completely on-device—both in the browser and natively on iPhone, iPad, and Mac.

Transcribing chess puzzles by hand or scraping unverified problem collections breaks down fast. Two failure modes show up immediately in real training:

1. Cooked solutions: A position has a flashy sacrifice, so a human annotator marks it as winning—missing a quiet defensive resource at depth 18, or missing a second equally winning move that punishes a student for playing an objectively strong alternative.
2. Horizon flips: A shallow engine pass (depth 10–12) flags a move as +2.10, only for depth 22 to reveal perpetual check (0.00).

Instead of transcribing puzzles, we built a deterministic mining and independent verification pipeline powered by Stockfish 18. Along the way, we hit a subtle FEN validation edge case in chess.js that silently breaks UCI engines.

Stage 1: Mining for Uniqueness, Not Just Blunders

A good tactics puzzle isn't simply "someone blundered in a game." The actual mathematical requirement for a fair puzzle is uniqueness: exactly one legal move must preserve or seize a decisive advantage, while every alternative drops significant win probability.

Our generator runs a multi-stage filter:

1. Human-strength self-play seeding: Stockfish plays games at calibrated Elo bands (1320–2500), randomizing among shallow top candidates in the opening so positions diversify across pawn structures rather than repeating five mainlines.
2. Shallow two-line screen (Depth 12, MultiPV = 2): Every candidate position is evaluated with two principal variations. Most positions either have multiple good moves or zero good moves; both are discarded cheaply here.
3. Deep uniqueness confirmation (Depth 20, MultiPV = 2): Survivors are re-searched at depth 20. We only keep a candidate if bestMove beats the second-best move by at least 140 centipawns (>= 140 cp) and achieves a concrete evaluation swing.
4. Branchless line extension: We step forward along the principal variation, extending the puzzle ply by ply only while every solver turn continues to have a single uniquely best reply. The moment two moves become interchangeable, the puzzle terminates cleanly.

That pipeline produced 14,351 verified tactics puzzles spanning 760–2899 puzzle rating, alongside 116 positional judgement exercises (where no forced tactical win exists) and canonical endgame drills.

Stage 2: Independent Depth-22 Verification & Time Caps

Never trust the same search depth that generated your dataset to grade your dataset. In scripts/verify-puzzles.mjs, we spin up a fresh Stockfish instance and re-verify every puzzle two plies deeper (depth 22) with a per-position wall-clock ceiling (6000 ms) so fortress-like positions cannot stall batch runs for hours:

const { lines } = await engine.analyse(puzzle.fen, { depth: DEPTH, // Default: 22 (two plies deeper than mining) movetime: TIME_CAP, // 6,000 ms ceiling for pathological positions multipv: 2, }); if (lines.length < 2) { problems.push('only one legal move'); } else { if (lines[0].move !== puzzle.solution[0]) { problems.push(`deeper search prefers ${lines[0].move} over ${puzzle.solution[0]}`); } const gap = cpFor(lines[0].score, solverColor) - cpFor(lines[1].score, solverColor); if (gap < 100) { problems.push(`solution is not unique (gap ${gap}cp)`); } } 

Why Endgames Need Time Budgets Instead of Fixed Depth

For endgame drills (rook vs. pawn, opposition, Lucena, Philidor, two bishops), fixed search depth fails in both directions:

• In a bare KR vs K position, depth 28 finishes in milliseconds.
• In a minor-piece endgame with pawns on both wings, depth 28 explodes combinatorially and times out.

Switching endgame verification from fixed plies to a fixed 4,000 ms movetime budget gave consistent tablebase-like stability across all drills, asserting cp >= 400 (or forced mate) on positions labelled "win" and |cp| <= 150 on positions labelled "draw".

The Illegal-FEN Trap chess.js Won't Catch

When curating hand-crafted endgame drills or mutating FEN strings in tests, there is one mistake that is shockingly easy to make: placing a piece so that the side not to move is already in check.

No legal sequence of chess moves can ever reach such a position—because the player who just moved would have left (or moved) their own king into check.

Here is the catch:

• new Chess(fen) in chess.js accepts these FENs without throwing.
• When you pass that same FEN to Stockfish over UCI (position fen ... -> go movetime 4000), Stockfish immediately responds with bestmove (none)—which looks identical to an engine crash or pipe timeout if you aren't checking for it!

The fix is a 10-line helper that flips the active turn token in the FEN, clears the en-passant target square, and asks chess.js whether the swapped side is in check:

import { Chess } from 'chess.js'; /** * True when the side *not* to move is in check — an illegal position * that no game can reach. chess.js accepts these, and Stockfish replies * with "bestmove (none)", which is easy to misread as an engine crash. */ function opponentInCheck(fen) { const parts = fen.split(' '); parts[1] = parts[1] === 'w' ? 'b' : 'w'; parts[3] = '-'; // Clear en-passant target before swapping active color try { return new Chess(parts.join(' ')).isCheck(); } catch { return false; } } 

Adding opponentInCheck(fen) before invoking UCI saved hours of debugging mysterious empty engine evaluations during endgame drill authoring.

Grading Human Moves: Win Probability vs. Raw Centipawns

Raw centipawn loss (CPL) is a poor teacher for human players because engine centipawns are non-linear:

In Brass Pawn's coached play and native iOS/macOS trainer (which pairs Stockfish 18 with Reckless on-device), moves are graded by win-probability loss rather than raw centipawn deltas. Trading down into a clean, slightly lower-centipawn king-and-pawn endgame isn't penalized as an "inaccuracy" when win probability stays pinned near 99%.

Architecture Snapshot: Cloud Chess Platforms vs. Local Engine Study

Because the full puzzle set and neural engine run locally, there is zero network round-trip per ply and zero account wall:

Try the Code or Run Your Own Verification Pass

• Open-Source Repo (AGPLv3): github.com/minchopm/chess-trainer (npm install && npm start, or inspect scripts/verify-puzzles.mjs)
• Live Web & Native App Overview: brasspawn.com (also available natively on iPhone, iPad & Mac)
• Technical Chess Guides & Trainer Benchmarks: brasspawn.com/guides and brasspawn.com/compare

If you maintain chess tooling or puzzle generators in JavaScript or Swift, drop opponentInCheck() into your FEN ingestion tests—it catches broken test fixtures before they ever reach UCI.