How I Fine-Tuned a 7B LLM with LoRA and Unsloth When I was working on Khatta-ka-LLM, two technologies became especially important: LoRA + Unsloth If you've heard these names but aren't sure what they actually do, here's a simple explanation. What is LoRA? LoRA stands for Low-Rank Adaptation. Instead of updating all the parameters of a large language model during fine-tuning, LoRA keeps the original model weights frozen and trains a much smaller set of additional parameters. Think of it like this: Full fine-tuning Entire model ↓ Update billions of parameters LoRA Original model ↓ Keep the original weights ↓ Train small adapter matrices The result is a much more parameter-efficient fine-tuning process. This is especially useful when you don't have access to expensive GPU clusters. Where does Unsloth come in? Unsloth is a framework designed to make LLM fine-tuning more efficient. It focuses on reducing memory usage and improving training speed, making techniques such as LoRA and QLoRA much more practical. For my Khattak Pashto experiment, my workflow was essentially: Qwen2.5-7B ↓ LoRA fine-tuning ↓ Unsloth optimization ↓ Custom Khattak Pashto dataset ↓ Fine-tuned model The goal wasn't to teach the model language from scratch. The base model already had broad language knowledge. Instead, I wanted to adapt that existing knowledge toward a specific regional dialect by exposing it to carefully prepared examples. What happened during training? For my experiment, the training loss went from: 3.44 → 0.22 That doesn't mean the model became perfect. Training loss alone isn't enough to prove that a model performs well in real-world usage. Proper evaluation and testing are still important. But it was a great demonstration of what parameter-efficient fine-tuning could achieve for this experiment. Why LoRA is interesting The important idea behind LoRA is that you don't necessarily need to modify every parameter of a large model to adapt it to a new task or domain. You can keep the original model and learn a relatively small set of trainable parameters. That makes experimentation significantly more accessible. For example, instead of thinking: «"I need a massive GPU cluster to fine-tune an LLM."» You can start thinking: «"What is the smallest efficient fine-tuning approach that can solve my problem?"» That's a very different way of approaching LLM engineering. The bigger lesson You don't always need massive infrastructure to experiment with LLMs. Understanding concepts such as: Parameter-Efficient Fine-Tuning (PEFT) LoRA QLoRA Quantization Efficient training frameworks can make LLM experimentation much more accessible. That's one of the things I enjoyed most about building Khatta-ka-LLM. The process was simple: Build ↓ Experiment ↓ Measure ↓ Improve If you're learning LLM fine-tuning, I'd recommend starting by understanding what LoRA is actually changing inside the model. Once that clicks, the rest of the workflow becomes much easier to understand. What's next? I can break down either: LoRA vs QLoRA or How LoRA works under the hood mathematically. Which one would you like to see next?