A few months ago, I started building a Retrieval-Augmented Generation application because I wanted to understand what was actually happening underneath all the buzzwords. The project I built was appropriately named simple_rag. The idea was simple enough: upload a PDF, ask questions about it, and have an LLM answer using information from the document. I got it working. Then I mostly abandoned it. The thing I couldn't figure out how to handle cleanly was running the embedding model locally. I accidentally made a simple project expensive My original implementation used Gemini for both embeddings and generation. Later, I switched it to Mistral. Both worked, but the more I played with the project, the more I disliked one part of the architecture: every time I wanted to turn some text into embeddings, I was calling somebody else's API, with its own rate limits and its own costs. That meant another API key, another service to depend on, another rate limit to think about, and potentially another bill. For a production application where the trade-off makes sense, fine. For a tiny side project whose entire purpose was basically: Antony would like to understand RAG. ...it felt ridiculous. Embeddings are fundamental to how this implementation works. When you upload a document, the text gets split into chunks, and each chunk gets converted into an embedding: a numerical representation of its meaning. That embedding gets compared against the question's embedding, and whichever chunks are most semantically similar get sent to the LLM as context. That is basically the retrieval part of Retrieval-Augmented Generation. So if creating embeddings depended on an API I didn't want to keep paying for, the whole experiment got a lot less interesting. The repository sat there. Then I discovered I was overcomplicating the problem Recently I came back to the project and discovered fastembed, and had one of those slightly embarrassing engineering moments where you realize the problem that made you abandon something months ago never actually needed the solution you'd assumed it needed. I could generate the embeddings locally, on my CPU, without touching an embedding API, a GPU, or a heavy PyTorch install, and without a per-embedding bill. I'm currently using BAAI/bge-small-en-v1.5 through FastEmbed. It runs on ONNX Runtime and is small enough that my completely ordinary laptop handles it without breaking a sweat. That changed the architecture quite a bit. The current flow is basically: PDF / EPUB ↓ Extract text ↓ Split text into chunks ↓ Generate embeddings locally on my CPU ↓ Store the vectors in memory ↓ User asks question ↓ Embed question locally ↓ Compare it with document embeddings ↓ Retrieve the most relevant chunks ↓ Send those chunks + question to the LLM ↓ Answer That's it. And weirdly, stripping things away helped me understand RAG much better than adding more AI tooling ever did. I don't even have a vector database This was another thing I had mentally associated with RAG. Embeddings mean vectors. Vectors mean vector databases. Therefore RAG means I need Pinecone, Qdrant, Chroma, pgvector, or something similar. Except... not necessarily. My application currently stores embeddings as plain Python data and uses NumPy to calculate cosine similarity. For every question, it compares the question's vector against the chunk vectors, sorts by similarity, and grabs the best matches. Is that how I'd build a system with millions of documents? Absolutely not. Is it enough for a PDF, an EPUB, or a few hundred pages while I'm trying to understand retrieval? Absolutely. That distinction has become increasingly important to me when learning new technology. There's a huge difference between asking what do I need to understand this, and what would I need to operate this at serious production scale. The internet has a habit of giving you the second architecture while you're still trying to answer the first question. The LLM is now the only remote part I still needed a model to generate the final answer, and this is where I found Groq particularly interesting. For my current side-project level of usage, their free tier is enough to experiment with. I created an API key without even adding a credit card, pointed the OpenAI-compatible client at Groq, and that was basically it. I'm currently using openai/gpt-oss-120b. So now the parts have a much cleaner separation. My machine handles: Extracting the document text. Chunking it. Creating embeddings. Storing those embeddings. Embedding the question. Retrieving the relevant chunks. The LLM handles: Taking the question plus retrieved context. Producing a useful, human-readable answer. That separation made another thing click for me: RAG isn't really about making the LLM know your data, since the model doesn't suddenly learn my PDF. I'm building a retrieval system that finds useful information first, then handing that information to the model at generation time. The retrieval is doing a lot more of the work than I originally appreciated. I wanted the implementation to stay boring The core RAG logic is only a few hundred lines of Python. That's deliberate. There's a FastAPI API around it and a small browser UI, plus a CLI because apparently I cannot make a Python project without eventually wanting to run it from the terminal. But the actual interesting bit is separate from all that. The RAG module handles: Text extraction. Chunking. Embedding. Similarity search. Retrieval. Generation. The interfaces just call it. I also added an evaluation script using a paper I already had lying around, Attention Is All You Need, with questions of varying difficulty. Again, the point was not to build some enterprise RAG platform. I wanted something small enough that I could open rag.py, read it top to bottom, and understand what every part was doing. I learn better that way. Local AI is becoming much more interesting to me The embedding discovery also changed how I think about adding AI features to my own applications. My default assumption used to be that "adding AI" meant sending everything to some massive remote model. That assumption is becoming less useful. There are increasingly parts of the pipeline I can run locally or self-host: Embeddings. Classification. Reranking. Transcription. Smaller language models. Various forms of semantic search. That leaves paid inference for the bits that actually benefit from a larger model, which matters for side projects, and matters even more when building software in places where throwing $50 or $100 a month at every interesting API isn't a particularly attractive development strategy. I already like self-hosting normal infrastructure. Apparently I'm slowly becoming the person who wants to self-host chunks of the AI stack too. This was probably inevitable. More importantly, I want to actually use this stuff I have zero interest in adding an AI button to every application just because investors have collectively decided that everything now needs AI. What I am increasingly interested in is finding places where these models genuinely improve an existing workflow. RAG is interesting because there are obvious uses for it. Documentation is one: imagine an internal application with years of manuals, procedures, policies and reports. Instead of manually searching through them, you ask a question and the relevant sections get retrieved before the answer is generated. That's useful. Another thing I want to experiment with next is databases. I've spent years building systems where somebody eventually asks: Can you give me a report showing X, grouped by Y, for these dates? ...and then somebody has to write SQL, add a report screen, export something to Excel, or build yet another dashboard. I want to experiment with a safer natural-language interface where somebody can ask: How many registrations did we receive per month this year? and the system understands the database schema, generates a read-only query, validates it, executes it, and explains the result. That's not necessarily the same problem as RAG. Structured databases need different techniques, and I definitely do not want an LLM firing random DELETE statements at production. But it's exactly the kind of AI experimentation I'm interested in now: giving the application another interface to capabilities it already has. The useful lesson wasn't actually about RAG I started this project because I wanted to understand embeddings, retrieval and RAG. I did learn those things, but I think the more useful lesson was about how easily we confuse the infrastructure around a technology with the technology itself. You do not need a vector database to understand vector search. You do not need an embedding API to create embeddings. You do not need an enormous framework to understand RAG. And you definitely do not need to start by building the architecture meant to serve ten million users. Sometimes you need a Python file, NumPy, a small model running on your CPU, and enough curiosity to keep poking at the thing until it stops looking like magic. That is what simple_rag has become for me. The repository is still called simple RAG. This time, I think it actually deserves the name. The project is on GitHub: tony-nyagah/simple_rag. I am going to keep experimenting with practical AI integrations and documenting what I learn, especially the boring, useful stuff between "call an LLM" and "build an autonomous superintelligence."