Building a Zero-Cost Local RAG Pipeline using Python and Ollama

dev.to Python2026-11-05 04:410 views

Retrieval-Augmented Generation (RAG) lets you query your custom private documents using large language models (LLMs) without sending sensitive data to third-party APIs.

In this tutorial, we will build an entirely local, zero-cost RAG pipeline using Python, Ollama, and ChromaDB.

Prerequisites

Python 3.9+ installed
Ollama installed locally (ollama pull mistral)

Step 1: Install Required Dependencies

Install ChromaDB (vector database) and the Ollama Python client:

pip install chromadb ollama

Step 2: Initialize Vector Database & Embeddings

We will store our knowledge base documents as vector embeddings in ChromaDB:

import chromadb import ollama # Initialize local persistent vector store client = chromadb.PersistentClient(path="./rag_db") collection = client.get_or_create_collection(name="local_knowledge") # Sample document knowledge base documents = [ "Our enterprise platform policy requires JWT rotation every 24 hours.", "System maintenance occurs every Sunday from 2 AM to 4 AM UTC.", "Database read replicas are deployed across US-East and EU-Central regions." ] # Generate embeddings locally via Ollama and store in ChromaDB for i, doc in enumerate(documents): response = ollama.embeddings(model="mistral", prompt=doc) collection.add( ids=[f"doc_{i}"], embeddings=[response["embedding"]], documents=[doc] ) print("Knowledge base successfully embedded and indexed locally!")

Step 3: Querying the Local RAG Pipeline

Now, retrieve context relevant to a user prompt and generate an accurate response:

def query_rag(user_query: str): # 1. Embed the incoming question  query_embedding = ollama.embeddings(model="mistral", prompt=user_query)["embedding"] # 2. Query top 2 matching document chunks  results = collection.query( query_embeddings=[query_embedding], n_results=2 ) retrieved_context = " ".join(results["documents"][0]) # 3. Pass context and question to the local LLM  prompt = f""" Answer the question based ONLY on the context below: Context: {retrieved_context} Question: {user_query} """ response = ollama.generate(model="mistral", prompt=prompt) return response["response"] # Example execution answer = query_rag("When does system maintenance happen?") print("AI Response:\n", answer)