Most "I built an LLM" posts show you a loss curve going down and stop there.

This one is different, because my from-scratch model lost, and I think the why is the most useful thing I can teach you.

Over the last few weeks I built two language models on the same task (empathetic, supportive dialogue) and compared them head to head:

  1. A GPT-style transformer trained from random initialization: 253M and 757M parameter versions, written in PyTorch, trained on my own hardware.
  2. Qwen2.5-0.5B-Instruct fine-tuned with QLoRA, in two stages, then merged back into a single fp16 checkpoint.

Around both I built a serving layer (FastAPI, OpenAI-compatible endpoints, Docker, Prometheus) and a verification harness so every number in this post can be reproduced from the repo.

Repo: github.com/bharqav/SLM

By the end of this article you'll understand:

  • how a decoder-only LLM works, from tokens to logits to sampled text
  • how I built and trained one, including the data mistakes that wrecked my first runs
  • what QLoRA is, with the actual arithmetic
  • why a 0.5B pretrained model beat my 757M scratch model, and why that isn't the paradox it looks like
  • what I got wrong, measured wrong, and had to correct along the way

Let's start with the machine itself.

Part 1: How an LLM actually works

Strip away the hype and a language model does exactly one thing:

Given a sequence of tokens, produce a probability distribution over the next token.

Everything else (chat, code completion, "reasoning") is that one operation applied repeatedly. Here's the full pipeline.

Step 1: Text becomes tokens

Neural networks eat numbers, not strings. A tokenizer splits text into chunks called tokens and maps each to an integer ID.

My scratch model uses GPT-2's byte-pair encoding (BPE) via tiktoken, with a vocabulary of 50,257 tokens. Qwen uses its own tokenizer with a vocabulary of 151,936.

import tiktoken enc = tiktoken.get_encoding("gpt2") ids = enc.encode("I have been feeling really down lately") print(ids) # a list of integers print(enc.decode(ids)) # back to the original text

BPE starts from bytes and repeatedly merges the most frequent pairs, so common words become single tokens and rare words get split into pieces. That's why a model can handle any string, including made-up words and code.

Step 2: Tokens become vectors

Each token ID indexes a row in a giant lookup table called the embedding matrix. For my 757M model that's 50,257 x 1,536, about 77 million parameters, just to turn IDs into vectors.

But there's a problem. Attention (coming up next) doesn't know word order by itself. "Dog bites man" and "man bites dog" would look identical. So we also inject position information.

  • My GPT-style model uses learned positional embeddings: another lookup table, 256 x 1,536, added to the token vectors.
  • Qwen uses RoPE (rotary position embeddings), which rotates query/key vectors by position-dependent angles instead of adding a learned table. It extrapolates to longer contexts far more gracefully.

Step 3: The transformer block

The model is a stack of identical transformer blocks. Mine has 24 of them. Each block has two sub-layers wrapped in residual connections:

x = x + Attention( LayerNorm(x) ) x = x + MLP( LayerNorm(x) )

Two ideas hide in those two lines.

Residual connections (x = x + ...) mean each layer only has to learn a correction to what's already flowing through. Gradients also get a clean highway back to early layers, which is what makes 24+ layer networks trainable at all.

Pre-norm (normalizing before each sub-layer, not after) keeps activations well-scaled and training stable, especially as depth grows.

Self-attention: tokens talking to each other

For every token, attention asks: which earlier tokens matter for predicting what comes next?

Each token's vector is projected into three vectors:

  • Query (Q): "what am I looking for?"
  • Key (K): "what do I contain?"
  • Value (V): "what do I pass along if you attend to me?"

Attention scores are softmax(Q @ K^T / sqrt(d)), and the output is the score-weighted sum of the values. Here's the heart of my implementation, simplified:

class CausalSelfAttention(nn.Module): def __init__(self, n_embd, n_head, dropout): super().__init__() self.n_head = n_head self.c_attn = nn.Linear(n_embd, 3 * n_embd) # Q, K, V in ONE matmul self.c_proj = nn.Linear(n_embd, n_embd) # output projection self.dropout = dropout def forward(self, x): B, T, C = x.shape q, k, v = self.c_attn(x).split(C, dim=2) # reshape to (batch, heads, time, head_dim) q, k, v = (t.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) for t in (q, k, v)) # is_causal=True applies the lower-triangular mask y = F.scaled_dot_product_attention( q, k, v, is_causal=True, dropout_p=self.dropout if self.training else 0.0, ) y = y.transpose(1, 2).contiguous().view(B, T, C) return self.c_proj(y)

Two details matter here.

Causal masking. During training, the model sees a whole sequence at once, so we must stop token t from peeking at token t+1, or it would just copy the answer. is_causal=True applies a lower-triangular mask so each position only attends to itself and the past.

Multiple heads. With 1,536 dimensions and 24 heads, each head works in a 64-dimensional subspace. Different heads can specialize: one tracks bracket matching, another subject-verb agreement, another something no human has labeled. We run them in parallel and concatenate.

The MLP: where tokens think alone

Attention mixes information between tokens. The MLP then processes each token's vector independently: expand to 4x the width (1,536 → 6,144), apply a nonlinearity (GELU in my model), project back down. A large share of a transformer's parameters, and much of its stored "knowledge", live in these MLPs.

Step 4: Vectors become a distribution

After the final block and a last LayerNorm, each position's vector is multiplied by the embedding matrix again, this time in reverse, to produce one score (a logit) per vocabulary token. Softmax turns logits into probabilities.

Sharing the same matrix for input embedding and output projection is called weight tying. It saves ~77M parameters and acts as a regularizer, because the embedding space has to be useful for both reading and predicting.

Step 5: Training is just next-token prediction

Training data is a long stream of tokens. For every position, the target is simply the next token. The loss is cross-entropy between the predicted distribution and that target:

logits = model(x) # (B, T, vocab) loss = F.cross_entropy( logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-100, ) loss.backward() optimizer.step()

That's it. No labels beyond the text itself. This is why LLM pretraining scales: the internet is the dataset.

A handy way to read the loss: perplexity = exp(loss). My best 24-layer empathy checkpoint hit a validation loss of 3.9259, which is a perplexity of about 50.7. Loosely: the model is as uncertain as if it were choosing between ~51 equally likely tokens at each step.

Step 6: Generation is sampling in a loop

To generate text, you feed in a prompt, get a distribution over the next token, sample one, append it, and repeat. How you sample matters enormously:

def sample_next(logits, generated_ids, temperature=0.7, top_k=20, rep_penalty=1.2): # penalize tokens we've already produced for t in set(generated_ids): logits[t] = logits[t] / rep_penalty if logits[t] > 0 else logits[t] * rep_penalty logits = logits / temperature # <1 = sharper, >1 = flatter topk_vals, topk_idx = torch.topk(logits, top_k) # keep only the k best probs = F.softmax(topk_vals, dim=-1) return topk_idx[torch.multinomial(probs, 1)]
  • Temperature rescales confidence. Low values give safe, repetitive text; high values give creative and sometimes unhinged text.
  • Top-k discards the long tail of unlikely tokens.
  • Repetition penalty fights the classic small-model failure of looping on the same phrase.

A note on the KV cache

Notice something wasteful: when generating token 101, the keys and values for tokens 1-100 haven't changed. A KV cache stores them so each new token costs one token's worth of compute instead of re-processing the whole prefix.

Qwen (via Hugging Face) uses a KV cache. My from-scratch model does not, and as you'll see in the benchmarks, that single omission matters a lot.

Part 2: Building the from-scratch model

The architecture

I deliberately followed GPT-2's design so every piece would be something I could fully explain. Two sizes:

  • 253M model: 16 layers, 16 heads, 1,024-dim embeddings
  • 757M model: 24 layers, 24 heads, 1,536-dim embeddings (about 680M non-embedding parameters)

Shared settings: 256-token context, 4x MLP expansion, GELU, pre-norm LayerNorm with bias, dropout 0.1, tied embeddings, learned positions, and gradient checkpointing, which trades extra compute for roughly 40% less activation memory.

One initialization detail worth knowing, straight from the GPT-2 paper. The output projections inside each block (c_proj) are initialized with a smaller standard deviation, 0.02 / sqrt(2 * n_layer). Every block adds to the residual stream, so without this scaling the stream's variance grows with depth and early training gets chaotic.

You can sanity-check the parameter count with back-of-envelope math: each block has roughly 12 x d^2 parameters (4d² for attention's Q/K/V/output, 8d² for the MLP). For d = 1,536 that's ~28.3M per block, x 24 blocks = ~680M. Add the 77M embedding table and you land right at ~757M.

Training run 1: the humbling CPU phase

My first pretraining ran on CPU only, float32, 8 threads, at about 215 tokens per second on a ~2.5M-token code-and-prose corpus. Two runs totaling roughly 2h44m got the 253M model to a best validation loss of 4.2803.

That number is not good. It's the kind of model that produces fluent-looking nonsense. But it was a complete, working end-to-end loop, and everything after it built on that foundation.

Training run 2: the GPU empathy probe

Next I trained the 24-layer, 757M configuration on empathetic dialogue (EmpatheticDialogues + ESConv, about 2.1M train tokens) on an RTX 5060 with bfloat16 mixed precision. Effective batch: 32 sequences x 256 tokens = 8,192 tokens per step.

Validation loss over time:

  • step 200: 4.8194
  • step 400: 4.4651
  • step 600: 4.1942
  • step 800: 4.0373
  • step 1000: 3.9259 (best, perplexity ~50.7)

Good progress. But when I actually read the outputs, something was badly wrong.

The data bug that taught me the most

The model's replies were non-sequiturs. It would answer "I feel like nobody understands me" with something fluent and completely unrelated. The loss curve looked healthy. The behavior was broken.

Digging into the corpus, I found two structural defects:

  1. No speaker markers. Turns were joined with a bare newline, so the model never learned which side of the conversation it was writing.
  2. Consecutive same-role turns. There were 2,511 runs of back-to-back user (or back-to-back assistant) messages, which taught the model that one message can be followed by an unrelated one.

Lesson: a falling validation loss only tells you the model is getting better at the distribution you gave it. It says nothing about whether that distribution was the right one.

The fix was a new preprocessing script that rebuilds the corpus with an explicit format:

User: <text> Assistant: <text>

with consecutive same-role turns merged, Alpaca instruction/response pairs added in the same format (so the model can answer general questions, not just offer support), over-long answers dropped, and a 1M-token slice of code appended raw. The model tells the two modes apart by prompt shape: a bare code prefix continues as code, a User:/Assistant: prefix continues as dialogue. Dialogue and Alpaca splits are disjoint, so validation is leak-free.

The clean corpus came to ~5.93M tokens: 22,364 dialogues, 44,481 QA pairs, and the 1M code tokens.

I also used --init-from to load only the model weights and reset the optimizer. When your data distribution changes, resuming old optimizer state means carrying momentum tuned for a different dataset.

At inference there's a matching trick. The model happily keeps writing after it finishes its reply, inventing the next User: turn. So I cut the generated text at the first User: or Assistant: marker after the answer, and train with an EOS token at the end of each sequence so sampling can stop cleanly.

Distilling Qwen into the small model

Even with clean structure, the small model had to invent its own style from very little data. So I tried distillation: run Qwen2.5-0.5B-Instruct over 20,000 Alpaca instructions (temperature 0.3), then mix those answers into the training corpus. Now the 253M model learns to imitate a stronger teacher's answers.

Validation loss across these runs on the clean corpus fell from 4.0473 to 3.2289 (the CLI/API default checkpoint, at step 8,000) and to 2.9197 for the later v3 checkpoint.

An important caveat: those numbers are not comparable with the 3.9259 from the empathy-only run. Different corpus, different validation set, different task mix. Validation loss is only comparable between models evaluated on the same data with the same tokenizer. I'll come back to this, because it nearly misled me.

Part 3: Why I fine-tuned Qwen

At this point I had a working scratch pipeline and a decent understanding of the whole stack. So why bring in Qwen?

Because the honest question was: what would it take to build the best possible empathetic assistant under my constraints?

Here's the gap I was up against.

Scale of pretraining. My scratch model saw millions of tokens. Qwen2.5 was pretrained on trillions. That's several orders of magnitude more data, and that's where grammar, world knowledge, code ability, and instruction-following come from. Fine-tuning on 23K dialogues can steer a model with all that capability. It can't conjure it.

Architecture upgrades. Qwen isn't just "bigger GPT-2". Its design (0.5B parameters, 24 layers, hidden size 896) includes:

  • RoPE positions instead of a learned table
  • RMSNorm instead of LayerNorm
  • Grouped-query attention: 14 query heads but only 2 key/value heads, which shrinks the KV cache dramatically
  • A gated MLP (896 → 4,864 → 896)
  • A 151,936-token vocabulary that handles many languages and code efficiently

Fit for the size bracket. At 494M parameters, Qwen2.5-0.5B-Instruct sits between my 253M and 757M scratch models. Same order of magnitude, so the comparison is meaningful, even if it isn't size-parity.

And importantly, I framed this correctly in the repo: it is not a claim that one model is a distilled or compressed version of the other. It's a side-by-side of two real training pipelines on the same task.

Fine-tuning without a data center: QLoRA

Full fine-tuning updates every weight, which needs optimizer state (several times the model size) in GPU memory. QLoRA makes it fit on a consumer card with two ideas.

1. LoRA (Low-Rank Adaptation). Freeze the original weight matrix W and learn a small update expressed as the product of two skinny matrices:

W' = W + (alpha / r) * B @ A

where A is r x d_in and B is d_out x r, with a tiny rank r. Instead of training a 896 x 896 matrix (802,816 values), a rank-8 adapter trains 8 x 896 + 896 x 8 = 14,336 values.

2. Quantization. Store the frozen base weights in 4-bit NF4 format with double quantization, and compute in bf16. The base model shrinks to a fraction of its memory footprint, while the small adapter trains in higher precision.

Let me show the arithmetic on my adapter, because it's a satisfying check. Qwen's q_proj maps 896 → 896 and v_proj maps 896 → 128 (2 KV heads x 64 dims). At rank 8, per layer:

  • q_proj: 8 x (896 + 896) = 14,336
  • v_proj: 8 x (896 + 128) = 8,192
  • per layer: 22,528, and x 24 layers = 540,672 trainable parameters

That matches the 540,672 scalars in my saved adapter exactly, about 0.11% of the 494M base model, and the whole adapter file is ~2.1 MB.

Stage 1: empathy SFT

  • Data: ~23,427 training dialogues / 461 validation dialogues (EmpatheticDialogues ~23K exchanges + ESConv ~910 multi-turn support dialogs), hash-split by dialogue to prevent leakage
  • Setup: 4-bit NF4 base, bf16 compute, AdamW at 1e-4 with warmup and cosine decay, batch 1 with 16 gradient-accumulation steps, 1,024-token sequences, 2 epochs
  • Cost: 2,928 optimizer steps in about 3.6 hours
  • Result: final train loss 2.1938; validation loss 2.7739 after epoch 1 and 2.7650 after epoch 2

Stage 2: a small steering pass

A second, much smaller pass used 770 rows of hand-written steering dialogues mixed with replay data from stage 1: 96 steps, 175 seconds, learning rate 5e-5.

Here's where I have to be honest, because the repo is.

Stage 2 did not measurably improve validation loss. Stage 1 ended at 2.7650; stage 2 at 2.7655. And the stage-2 validation set was the same 461-row file used in stage 1, so it isn't an untouched hold-out anyway. Validation loss was never going to capture what steering is for (policy and behavior), but I also have no automated behavioral gate in the repo that proves it worked. My stored behavioral probe records predate the latest retrain and no script reproduces them. I've listed that as a known gap rather than pretend otherwise.

I also hit a subtle config surprise: the stage-2 run was launched with --init-adapter, and my training script ignores --lora-r and friends in that path, loading the stage-1 config instead. So the trained stage-2 adapter is rank-8 over q_proj/v_proj, even though its log records the CLI values (r=16, alpha=32). Always verify what your adapter config actually is, not what you typed on the command line.

Merging

For deployment I merge the adapter back into the base weights and save a single fp16 checkpoint (988,097,536 bytes). The merged model loads as a plain AutoModelForCausalLM: no PEFT wrapper, no adapter step at load time, and zero extra inference cost, since the architecture is identical to the base.

model = AutoModelForCausalLM.from_pretrained( "slm-ft/merged_fp16/", torch_dtype=torch.float16, device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained("slm-ft/merged_fp16/")

Part 4: Head-to-head results

All benchmarks: CPU, float32, 8 threads, identical prompts, max_new_tokens=40, temperature 0.7, top-k 20, top-p 0.9, repetition penalty 1.2, seed 1337. Reproducible from benchmarks/generation_matrix.json.

Speed and memory

Generation throughput:

  • SLM 16L (253M): 12.54 tok/s
  • Qwen2.5-0.5B-Instruct (base): 9.92 tok/s
  • Qwen + QLoRA merged: 10.62 tok/s
  • SLM 24L (757M): 4.69 tok/s

Memory after load (RSS):

  • SLM 16L: 2,907 MB
  • Qwen base: 3,631 MB
  • Qwen merged: 3,923 MB
  • SLM 24L: 6,078 MB

Reading these correctly:

  • The 16L model is genuinely faster than Qwen: fewer parameters, so less memory to stream per token. It's 1.26x faster at generation and 2.5x faster per forward pass.
  • The 24L model is 2.1x slower than Qwen despite having only ~1.5x the parameters. The reason is the missing KV cache. It re-encodes the entire context at every generated token, while Qwen reuses cached keys and values. This is the single most actionable finding for anyone building their own model: implement the KV cache.
  • Don't over-read the merged-vs-base gap. The merged model measured 7% faster than base in the final run, but across earlier runs of the same script the direction reversed (-11% and -6%). Same architecture, same FLOPs, so per-token cost should be identical. I treat that gap as unmeasured noise, not a result. CPU timings on my machine vary by roughly 10% run to run, so the ratios are the finding, not the absolutes.

Quality

Speed is where my scratch models can compete. Quality is where they can't.

  • SLM 24L: On the empathetic prompt it stays on topic in a stilted, all-lowercase register ("i am sorry to hear that, but i hope you get through it soon..."). On code prompts it falls apart: def preprocess(text): produced "You have an appointment?" and class ChatMessage: produced nothing at all. That's the predictable result of training only on empathy data.
  • SLM 16L: Short, on-topic empathetic replies ("That is good to hear! I hope everything goes well for you."). On code prompts it produces prose or fragments, not code.
  • Qwen2.5-0.5B-Instruct: Coherent and on-topic on every prompt, including syntactically valid code completions for all four code prompts.
  • Qwen merged fine-tune: Same behavior as the base on all five benchmark prompts. The good news: the narrow empathy SFT did not wreck its code ability, which is the main regression risk of fine-tuning on a small, specialized corpus. Whether it improves empathy still needs the behavioral gate I haven't built yet.

Bottom line: neither from-scratch model matches Qwen's output quality on these prompts. The project's quality-oriented path is the QLoRA fine-tune. The scratch models are the educational path, and the 16L one is a perfectly reasonable fast model for short supportive replies.

Why this isn't a paradox

A 757M model losing to a 494M one sounds backwards until you separate the three things that determine quality: parameters, data, and compute. My 757M model had more parameters but trained on roughly a million times less data than Qwen saw, for a few thousand steps on a single consumer GPU. Parameters are capacity; data and compute are what fill that capacity. A large empty model is just a large empty model.

That's the real reason fine-tuning wins at small scale: you rent someone else's trillions of training tokens and spend your own tiny budget only on the last mile.

Part 5: Serving it and verifying it

A model nobody can call is a notebook. So I built the serving layer properly.

FastAPI server (slm_api.py) with:

  • /generate plus OpenAI-compatible /v1/completions and /v1/chat/completions, with SSE streaming variants, so existing OpenAI clients can point at it
  • Bearer-token auth, per-IP rate limiting (HTTP 429), request timeouts (HTTP 504), and restricted CORS
  • Input and output safety guardrails that reject harmful prompts and scan generated text. For a model aimed at emotional-support conversations I wanted that layer to exist from day one
  • /health, /model/info, and a Prometheus /metrics endpoint tracking request counts, latency histograms, tokens generated, and model-loaded state

Deployment: a slim Docker image (CPU-only PyTorch wheels, because the default Linux wheel bundles several GB of CUDA runtime I wasn't going to use), non-root user, health checks, and a production docker-compose stack with Prometheus and resource limits.

Verification harness: one command, python harness.py, runs the test suite, a setup-artifact check, the live comparison, and a dependency audit, then exits non-zero if any step fails. There's also seeded generation (seed=1337), pinned dependency constraints, pre-commit hooks, and a held-out perplexity evaluation script.

I added the harness after a mistake, which brings me to the part I think matters most.

Part 6: What I got wrong (and fixed)

I kept a "provenance note" every time a number in my own docs turned out to be wrong. There were several:

  • A wrong validation loss. An earlier table said "step 500, val loss 3.9733" for the 24-layer model. That number belonged to a different run (a 458M, 20-layer model). The correct shipped result is step 1000, 3.9259. I only caught it by tracing each figure back to its log file.
  • A double-counted memory figure. My benchmark reported 5,616 MB for the merged Qwen model. The harness had left the base model resident in the same process, so both rows carried the base's ~2 GB. After freeing it, the real number is 3,923 MB.
  • A benchmark "result" that was noise. Merged vs. base speed flipped direction across runs. If I'd published only one run, I'd have "found" a speedup that doesn't exist.
  • An adapter config that didn't match its log, as described above.
  • A fine-tune that's "verified" only by loss. The behavioral gate is still missing.

None of these were catastrophic, because each got caught by the same habit: every number in the docs must trace to an artifact in the repo, and every claim must say what it does and doesn't prove. If you take one engineering lesson from this post, take that one. It's more valuable than any architecture trick.

Takeaways

  1. LLMs are simpler than they look. Embeddings, attention, MLPs, residuals, and a softmax, applied next-token-at-a-time. Building the whole thing once makes every paper easier to read.
  2. Data quality beats everything. My model looked healthy on the loss curve while giving non-sequiturs, and the cause was missing speaker markers and merged turns in the corpus.
  3. Validation loss is only comparable on the same data and tokenizer. Don't compare numbers across corpora.
  4. Implement the KV cache. It was the difference between "slightly slower than Qwen" and "2x slower."
  5. At small scale, fine-tune instead of pretraining. Parameters are capacity; pretraining data is what fills it. QLoRA let me adapt a capable model with 0.11% trainable parameters in an afternoon.
  6. Measure honestly. Run benchmarks more than once, label what's noise, free your memory between measurements, and write down what you didn't verify.

What's next

  • An automated behavioral evaluation for the merged model (the biggest gap)
  • A KV cache for the from-scratch model
  • Standard zero-shot benchmarks (HellaSwag, ARC, PIQA, and others) for a fairer capability comparison than my own prompt set

Try it yourself

Everything (the model code, training scripts, QLoRA pipeline, benchmarks, API server, Docker setup, and harness) is open source under the MIT license:

github.com/bharqav/SLM

License note: the training data includes EmpatheticDialogues (CC BY-NC-4.0) and ESConv (CC BY-NC-SA), so models trained on it carry non-commercial constraints. Check the repo's License section before using any weights commercially.

If you build your own, I'd love to hear what broke first. In my experience it's never the architecture; it's the data.

Questions or corrections? Drop a comment, or open an issue on the repo. If a number in this post doesn't match the artifacts, the artifacts win.