I have been running a long series of experiments on whether small models can be made to behave usefully, and one obvious thing to borrow is whatever TinyStories did. That paper trained very small transformers to write coherent children's stories, and the detail everyone remembers is the vocabulary: roughly 1,500 words. An earlier experiment of mine had already varied the domain at a fixed vocabulary, restricting a corpus to a single arXiv category, and found a real win: +3.7 points of top-1, perplexity down 1.8x, and 17x less degeneration into repetition. So the natural follow-up is the other axis. Hold the domain and vary the vocabulary. If the vocabulary is the lever, capping it should help. It does not help; it hurts, by 4 points, and it wrecks the generated text. The research repo is not public, so the figures below come from the experiment's own harness and its recorded results.json rather than from anything you can clone. The fairness problem, first A smaller vocabulary makes next-token prediction trivially easier, because everything outside the cap becomes <unk> and <unk> becomes a frequent, easy target. Any arm with a smaller vocabulary will look better on a naive score, and it will look better for a reason that has nothing to do with capability. So every arm is scored only on positions whose target is inside the smallest vocabulary, which is 1,498 types. Identical positions, identical targets, across all three arms; <unk> is never a scored target. 3,000 scored positions each. Getting that right was most of the work, and it is the part I would want someone to check first. Without it the headline result reverses, and the reversed version is the one that would have confirmed what I expected to find. The numbers Same corpus (846K training tokens of cs.CL abstracts), same model at 4 layers and d=256 with a 128-token context, same budget per arm. Vocabulary capped at 1,500, 4,000 and 8,000 types. vocab cap training <unk> rate top-1 top-5 MRR perplexity 1,500 18.5% 0.211 0.446 0.321 75.9 4,000 7.5% 0.251 0.460 0.349 72.9 8,000 3.2% 0.253 0.454 0.348 86.4 Cap 1,500 is 4 points behind on identical scored positions. There is no accuracy to buy here at all, which was not the result I was expecting to write down. The generation metrics are worse than the accuracy ones: vocab cap repeated 4-gram distinct-4 novel 4-gram <unk> in output 1,500 0.075 0.925 0.291 36.8% 4,000 0.050 0.950 0.557 15.0% 8,000 0.044 0.956 0.671 1.8% More than a third of the emitted tokens at cap 1,500 are the unknown-word placeholder. Novel 4-grams collapse from 0.671 to 0.291 while repetition rises. The model becomes more copy-like, which is precisely the opposite of the effect I was trying to reproduce. The samples make it concrete. At cap 1,500: we propose <unk> - guided <unk> , a multimodal adaptive multimodal <unk> framework for <unk> multimodal <unk> and <unk> <unk> analysis . At cap 8,000: we propose <unk> , a framework that combines long - horizon memory with a lightweight context model with adaptive memory . in this architecture , the model observes the user before the tool to a small fraction of the context … The second one is not good prose, and it is recognisably prose. The first is a grammatical skeleton with every content word deleted. Why capping a vocabulary is not simplifying That last observation is the whole finding, and it took the samples rather than the metrics to see it. TinyStories restricted the generative process. The stories were written using simple words, so every token in the corpus carries meaning, and a small model learning that corpus is learning a language in which short words are sufficient. Capping the vocabulary of hard text restricts only the representation. The content words are not simplified, they are deleted and replaced with a placeholder that means nothing, and what remains is grammar with nothing to say. So the two operations look similar written down as "small vocabulary" and are opposites in what they do to the data. One produces a corpus that is genuinely simpler; the other produces a corpus that is damaged, and the damage falls entirely on the words that carry the information. You cannot recover TinyStories' benefit post-hoc from difficult text. If you want that effect on your own corpus, you need text that was actually written simply, not a smaller softmax over text that was not. The part that is still useful There is a mild sweet spot at 4,000 types: the best perplexity of the three at 72.9, and a top-1 statistically tied with 8,000 at four times fewer types. Worth knowing if you are fitting a model into a footprint, because the embedding table scales with the vocabulary and that table is usually most of a small model's parameters. But that is a memory optimisation and not a capability one, and the two get conflated constantly because they are the same dial. Turning it down saves bytes; it does not make the model better at anything. What generalises The reusable part is not about vocabularies. It is that "restrict X" can name two operations that are opposites, depending on whether you restrict the process that generated the data or the encoding of data already generated. Domain restriction in the earlier experiment was the first kind: the corpus really was drawn from a narrower distribution. Vocabulary capping here was the second kind, and it read as the same sort of intervention right up until the samples were printed. The other reusable part is the scoring fix. When one arm of an experiment makes the task easier, the arm that wins is the one that made it easiest, and no amount of care elsewhere recovers that. Restricting the scored positions to the intersection cost about ten lines and it is the only reason these numbers mean anything.