Build an LLM From Scratch in 5 Minutes on a Free GPU: I Trained a 5.9-Million-Parameter Storyteller, and Here Is What It Wrote

Part 1 of ten. Each post is a short, plain-words tour of one lesson from e=mc², a free build-an-LLM-from-scratch course; the full lesson, notebook and numbers are on the site.

Most people think building a language model takes a lab, a budget and a team. Last month I built one from nothing, on a free Google Colab GPU, in about five minutes end to end. It has 5.9 million parameters, it learned to write bedtime stories, and it is sitting on the Hugging Face Hub right now where you can run it.

I want to show you what it wrote, what the numbers mean, and why doing this once makes every headline about trillion-parameter models stop sounding like magic.

Meet Momo and Ella

The lesson is told through two characters, because the ideas are simpler than the vocabulary.

Momo is a young monkey learning to tell stories. Inside his head are millions of tiny dials, all pointing somewhere random, so his first stories are nonsense.

Ella is an elephant, and elephants never forget. Every night she reads Momo a story one word at a time, stops before each word, lets him guess, and tells him how wrong he was. Then she nudges his dials a little. Thousands of nights later, Momo tells stories.

That is the whole of machine learning in one paragraph. Momo is the model. His dials are the weights (5.9 million of them here). Ella's stories are the training data. "How wrong were you" is the loss. Nudging the dials is gradient descent. Everything else is detail, and the lesson decodes about forty of those details one at a time.

What a language model does, in one sentence

Given the words so far, guess the next one. A story is that guess repeated. A chat answer is the same thing with your question as the words so far.

When Momo continues "Once upon a time, a little", four things happen. The sentence is bitten into tokens, pieces of text with a number on each (our vocabulary has 4,096). Each token becomes a spot on a meaning map, a list of 256 numbers where similar words sit close together. Then every token looks back at the tokens before it and decides how much to listen to each: "capital" finds "France" and borrows some of its meaning. That is attention, from the 2017 paper that started all this, and it runs six times in a row, four heads each. Finally the last token's sharpened meaning is scored against every word in the vocabulary, and one is picked.

That is it. That is also, with more stories, more blocks and more GPUs, what the big ones do.

The numbers, honestly

5.9 million parameters: 6 blocks, 4 attention heads, 256-wide, a vocabulary of 4,096 pieces learned from the stories themselves.
1,500 training steps, each a batch of 32 windows of 256 tokens.
1.2 minutes of training on Colab's free T4 GPU; about five minutes for the whole notebook; about two minutes on an Apple Silicon Mac.
Perplexity 9 on 200 stories the model never saw. Perplexity is "between how many words is Momo still choosing". Random guessing would be 4,096. Watching that number fall from 4,096 to 9 in front of you is the moment the field stops being mysterious.

The 200 quiz stories were held out before training, so nothing the model memorised can flatter the score. An earlier draft got that wrong, and fixing it is why I trust the 9.

What it wrote

Given "Lily had a red balloon.", one run produced this:

She ran to her mom and gave her a big bite. They both ate the balloon and had the balloon. It was shiny and yellow. They ran to the park to play with.

Grammatical, kid-flavoured, slightly unhinged. That is exactly what 5.9 million parameters and two minutes buy, and I think it is wonderful.

Then I gave it "Once upon a time, a little monkey", and it wrote this:

Momo was sad because he could not swim like Bo the bear cub. He sat in the tall grass with his feet in the water. Ella said, "Try again, little one. Every time you try, you learn a little more." She held him up with her trunk. Momo kicked and kicked. Soon he could swim all by himself. The end.

Lovely, right? It is also, character for character, the second half of one of the thousand Momo stories I wrote into the training data, which the model read three times. That is memorising, not inventing. A few thousand repeats are easy for 5.9 million dials to store. The lesson shows you how to tell the difference, and why the fix is more unique stories and fewer repeats. Every large model has this problem too; it is just harder to catch when the training set is the internet.

Run it in thirty seconds

You do not have to train anything to try it. The weights, tokenizer and the notebook that builds them are public:

from huggingface_hub import hf_hub_download from transformers import PreTrainedTokenizerFast import json, torch # Config, GPT and tell_story() are the ~80 lines from Parts 6 and 8 of the notebook cfg = json.load(open(hf_hub_download("singhpratech/tiny-storyteller", "config.json"))) state = torch.load(hf_hub_download("singhpratech/tiny-storyteller", "model.pt"), map_location="cpu") tok = PreTrainedTokenizerFast.from_pretrained("singhpratech/tiny-storyteller") model = GPT(Config(**cfg)); model.load_state_dict(state); model.eval() print(tell_story("Once upon a time, a little monkey", temperature=0.8))

Start prompts the way the stories start ("Once upon a time, a little monkey"). Temperature 0.8 is the default; turn it up and the stories get stranger.

Why start this small

Because once you have turned 5.9 million dials yourself and watched perplexity fall from 4,096 to 9, the big numbers stop being magic and start being engineering. The recipe does not change as it scales: tokenizer, embeddings, stacked attention, a next-word head, a loss, an optimiser. Only the zeros do.

Want to actually build it?

This post is the postcard. The lesson is the trip. Lesson 01 of e=mc² on theaivibe.org walks you through every piece with widgets you can poke, a notebook that runs on a free GPU in minutes, and the honest numbers at every step. No account, no paywall, nothing to install.