$ cargo tree stonefish_ai v1.0.0 That is the whole dependency tree. No PyTorch, no candle, no cudarc — I wrote
the autograd, the byte-level BPE tokenizer, the transformer and the CUDA kernels by
hand. The GPU driver, cuBLAS and NVRTC are loaded at runtime through the operating
system's own loader, so even the GPU path has no crates.
It works: 222,331,904 parameters, 2.05 billion tokens, held-out bits-per-character
of 0.6064 against a 2.909 random baseline.
But the interesting part was the bugs, because none of them crashed.
A web handler that throws gets fixed in an hour. A gradient that is subtly wrong
produces a slightly worse model — and a slightly worse model looks exactly like
"the model is just small".
Here are three.
1. The gradient that looked reasonable
The attention backward pass used q where it needed dOut. It produced a
correctly-shaped matrix of the right magnitude, and it was wrong.
The loss did not move for days. I blamed the learning rate, the initialisation,
the data.
The fix took one line. Finding it took a numerical gradient check: compare the
analytic gradient against a central-difference estimate of the same gradient, and
fail if the relative error is above a threshold. I now have 32 of those tests. The
failure mode I was accepting — "close enough" — is exactly what the test forbids.
2. beta = 1.0 when I meant "overwrite"
cuBLAS computes C = alpha·A·B + beta·C. I passed beta = 1.0, which means "add
to whatever was in C". Since C was a reused buffer, every step added to the
previous step's output.
The numbers were not garbage. They were plausible and drifting. A four-line test —
run the same input twice, compare — found it immediately.
3. The profiler that reported 0 ms
My profiler showed 0 ms for the two biggest line items. That is not speed, that is
a broken hook attached to a code path that had been refactored away.
The lesson: a profiler that reports zero is broken, not fast.
What actually helped
Read the output, not the status. A green build means it compiled.
Compare against a second implementation. I rebuilt the same architecture in
PyTorch and compared the loss on the same weights and batches. They agree to
1.8e-05. Without that number, "my model is fine" is a belief.
Write the failures down. The full list is 23 items, and it is longer than the
feature list.
Because there are no dependencies, the whole thing is 839 KB of code — against
4.11 GB for a PyTorch install. That is the trade I was after: not speed, not
quality, but being able to read every line that makes it run.
The code: https://github.com/Edipcm-hash/stonefish
The weights: https://huggingface.co/StonefishAI/stonefish-222m
cargo run --release -- demo 90 Trains a small model from scratch and writes Shakespeare in 90 seconds.

