Part 5 of ten. Each post is a short, plain-words tour of one lesson from e=mc², a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site.
Momo can finish a story, but until tonight he could not take a request. That gap, between a model that continues text and a model that does what you ask, is the gap between a base model and an assistant. The step that closes it is called instruction tuning, and in Lesson 05 of e=mc² I did it by hand on the 5.9-million-parameter storyteller from the earlier lessons. On 50 requests the model had never seen, the share of stories with the right character in the right place went from 12% to 44%. Here is the one idea behind that number, and what it does and does not mean.
Completion is not following
Every lesson so far has been the same game. Ella the elephant reads Momo the monkey a story one word at a time, stops, lets him guess the next word, and nudges his dials. Give him "Once upon a time" and he writes a story. That is completion.
But say to him "a story about Zuri being brave in a storm by the watering hole" and he does not know that was a request. He hears words and keeps going: handed 50 such requests as plain text, only 6 of his stories happened to land on both the character and the place.
Following is a different game: read a request, then write what it asks for. The surprising part is how little it takes to teach.
Two signs and a red pen
Ella changes two things about the lesson.
First, she holds up two signs. One goes up before she says the request; the other goes up when it is Momo's turn to tell the story. The signs are not words. In the real model they are two special tokens, written <|instruction|> and <|story|>, which the tokenizer never splits apart. Request, then story, always in that fixed order, is the chat template. Every assistant you have talked to has one.
One detail bit me: Momo's meaning map has one row per token, 4,096 of them, and the new signs are tokens 4,096 and 4,097. Those rows do not exist yet, so the map grows to 4,098 rows, and the tied scoring layer grows with it.
Second, Ella puts her red pen down during the request. She grades only the story. If she graded the request as well, Momo would spend his effort learning to predict what she was about to ask, which is useless. In code this is prompt masking: every position inside the request gets the label -100, the value the loss function skips. A widget on the lesson page shows one real pair: 132 of its 156 guesses are graded with the mask, all 156 without it.
That is the whole mechanism. Here is the code that does both:
INSTR, STORY = "<|instruction|>", "<|story|>" tokenizer.add_special_tokens({"additional_special_tokens": [INSTR, STORY]}) # ids 4096 and 4097 model = resize_vocab(model, len(tokenizer)) # 2 new rows; the tied output head grows with them def encode_pair(instruction, story): prompt = [eos_id, instr_id] + tokenizer(instruction)["input_ids"] + [story_id] ids = prompt + tokenizer(story)["input_ids"] + [eos_id] x = ids[:-1] y = [t if j + 1 >= len(prompt) else -100 for j, t in enumerate(ids[1:])] # -100: not graded return x, y What Ella read him
4,000 request-and-story pairs. Half came from the TinyStories-Instruct dataset, where each story carries a one-line summary and three words it must use; those became a request like "Write a story about this: (summary) Use the words a, b and c." The other 2,000 came from the Momo and Zuri story generators, which write their own requests:
Write a story about Zuri sharing the shade with Tumi in the long yellow grass.
Then 500 training steps in batches of 32: fifty-one seconds on an Apple Silicon Mac with the GPU shared, 37 in an earlier run.
The test is 50 requests, 25 about Momo and 25 about Zuri, each asking for a combination of character, scene and place that appears in no training story (a leakage cell confirms it). The score is a plain string match: does the story contain the character's name and the exact place phrase?
The numbers, honestly
- 12% to 44%: 6 of 50 held-out stories had both the character and the place before tuning, 22 of 50 after.
- Character 68% to 100%. After tuning, every story named the right character.
- Place 14% to 44%. The hard half.
- Familiar combinations 8% to 86%, on 50 requests whose character, scene and place had appeared together in training.
- Cross-world: 1 of 20. Asked to put a character in the other one's world, it almost never did.
- Held-out loss 2.540 to 2.145 on 150 pairs the model never trained on.
- Whole notebook: 140 seconds on the Mac in the final run (102 earlier). Not yet timed on a free Colab T4.
Momo does not understand a request. He learned that the words right after the first sign, a name or a place, are good words to copy into the story. The character is easy because it shows up in every pair. The place is hard because 840 of the 2,000 generator stories name no place at all, so only about 58% of those pairs taught place-copying. The typical miss has the right character and scene, set in a place he had practised for that scene: asked for Momo hiding from the rain near the little pond, he hid him in the tall grass.
Two more honesty notes. A string match cuts both ways: a muddled story that says "Zuri" and "by the watering hole" counts, and "near the pond" does not count for "near the little pond". And the Momo and Zuri stories are built from templates, so they are recited sentences rather than invention. The real test of following is a TinyStories-Instruct request: asked for a little dog who finds a lost ball in the park and gives it back to a girl, the model wrote a park, a ball and a girl named Lily, and no dog.
The 44% is real and held across three local runs. It is also a long way from "understands requests": it is Momo learning where to look for the words to copy.
Why this small step matters
Because it is the same step the big assistants took. A base model, trained only to continue text, becomes an assistant by being fine-tuned on request-and-answer pairs in a fixed template, with the request masked out of the loss. The frontier uses far more pairs, far bigger models and a second stage of human preference training on top. The mechanics do not change. Once you have grown an embedding by two rows and set a few labels to -100 yourself, "chat template" stops sounding like a product feature and starts sounding like what it is: a fixed order and two signs.
The full lesson has the glossary, the results table, the leakage cells, the token-mask widget and the tuned model with its 4,098-token tokenizer and chat template, ready to run.
Want to actually build it?
This post is the postcard. The lesson is the trip. Lesson 05 of e=mc² on theaivibe.org walks you through every piece with widgets you can poke, a notebook that runs on a free GPU in minutes, and the honest numbers at every step. No account, no paywall, nothing to install.
- Start here: https://theaivibe.org/emc2/instruction-tuning-from-scratch-tiny-storyteller
- The whole ten-lesson journey: https://theaivibe.org/emc2
Tomorrow: Lesson 06 does the same job again with LoRA, changing only 2% of the weights, and asks whether the result is as good.
