Imagine you want to build a system where users can ask your AI system anything about the company. A user asks "When will my money arrive?" This exact answer sits inside your company's payout policy page. However your AI cannot read every document in your database each time a user asks a question. Scanning thousands of pages for a single query will be too slow and expensive,and it can also exceed the model's processing limits.

The system must quickly locate the exact paragraph that covers transfer timelines and pass only the relevant parts to the model so it can write the answer. This approach is commonly known as Retrieval-Augmented Generation (RAG).

What is RAG?

RAG is a hybrid AI framework that connects a large language model (LLM) to your own external knowledge source, like policy documents or internal wikis, to improve the quality and scope of the response.
Building a complete RAG pipeline involves different stages, from ingesting raw files to generating the final user response. This guide focuses on the first 2 steps after you’ve collected your documents: chunking and embedding.
By the end of this guide, you will understand the exact role each step plays in making your documents searchable.

Prerequisites

  • Python 3.10 or newer. Check with python3 --version.
  • Basic Python knowledge, such as running a script and installing packages with pip.
  • A few GB of free disk space and an internet connection for the first run. The libraries are large (sentence-transformers installs PyTorch), and the embedding model downloads once, at around 130 MB.
  • Some text to practise on. I’m using sample documents for a fictional company. Any folder of .txt files works.
  • No GPU is needed. Everything here runs on a normal laptop CPU.

Set up a virtual environment and install the libraries:

python3 -m venv venv source venv/bin/activate # on Windows: venv\Scripts\activate pip install langchain-text-splitters sentence-transformers 

Chunking

You already have your company's public page contents saved as plain text files and you want to begin by chunking these documents.
Before we write any demo/code, let's be clear about what chunking actually is.
Chunking is the process of breaking large documents down into smaller, manageable pieces of text before storing them in a searchable database.

AI models have strict limits on how much text they can process accurately at one time, and search engines work best when matching a specific question against a focused piece of text. If you put your entire knowledge base into your database as a single block, the search system cannot accurately give out the exact paragraph the user needs when queried.

Why split documents carefully?

Getting the chunk size right is a very important factor in search and context.

Too big gives blurry matches. If a single chunk covers transfer limits, card fees, and account verification all at once, it loses meaning overall. When a user asks a specific question about payout times, the search match becomes blurry because that single answer is inside unrelated texts.

Too small loses context. Imagine a chunk that simply reads: "fees are capped at ₦2000." on it own, that sentence is incomplete and quite useless. It loses context and the AI might be forced to guess.

Choosing the right LangChain splitter

When people first build a RAG application, they often grab a splitter from a framework like LangChain without checking whether it fits their data. LangChain offers several built-in options, each built for a different kind of document:

  • CharacterTextSplitter: This is the simplest option. It splits the text on a single separator, such as a line break, then merges the pieces back together up to your size limit. The catch is that it only knows one separator. If a block doesn't contain it, that block stays in one piece even when it's longer than your limit (LangChain logs a warning when this happens).
from langchain_text_splitters import CharacterTextSplitter splitter = CharacterTextSplitter(separator="\n", chunk_size=600, chunk_overlap=0) 
  • MarkdownHeaderTextSplitter: This splitter is fantastic if your files are saved in markdown format. That is, documents with # and ## headings, because it groups text under its matching section title. Two things to know. First, it doesn't enforce a chunk size, so a long section stays one big chunk, and people usually pass its output to a recursive splitter afterwards. Second, by default it removes the headings from the text and stores them as metadata, so you'd have to put them back into each chunk yourself. Text extracted from PDFs rarely keeps clean markdown headings, so this splitter works best on files that we markdown to begin with.
from langchain_text_splitters import MarkdownHeaderTextSplitter splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "Title"), ("##", "Section")]) 
  • SemanticChunker: Instead of counting characters, this tool uses an embedding model to detect when the topic shifts between sentences. It works quite well, but the downside here is that every sentence has to be embedded just to decide where to cut, which is more work than well-formatted pages need. You might want to consider this for documents that are not well formatted.
from langchain_experimental.text_splitter import SemanticChunker splitter = SemanticChunker(embeddings_model) 


python

  • RecursiveCharacterTextSplitter: This works best for general text. Instead of blindly chunking at a fixed character count, it works through a hierarchy of natural separators.
from langchain_text_splitters import RecursiveCharacterTextSplitter MAX_CHARS = 600 splitter = RecursiveCharacterTextSplitter( chunk_size=MAX_CHARS, chunk_overlap=50, separators=["\n\n", "\n", " ", ""] ) def split_page(text: str) -> list[str]: return splitter.split_text(text) 

Which one to use?

Splitter Use it when Watch out for
CharacterTextSplitter Text is uniform and has one clear separator Oversized chunks when the separator is missing
MarkdownHeaderTextSplitter Your files are real Markdown No size limit; headings go into metadata
SemanticChunker Text is badly formatted, with no useful structure Slower and costlier; experimental
RecursiveCharacterTextSplitter General text, and the safe default Doesn’t add headings or metadata for you

The examples in this guide use sample documents I generated for a fictional company called Brightlane Pay, so the fees and policies you’ll see are made up.
Here is RecursiveCharacterTextSplitter running on the refunds page from those sample documents:

from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=600, chunk_overlap=50, separators=["\n\n", "\n", " ", ""], ) pieces = splitter.split_text(body) # body = the text of the refunds page print(f"{len(pieces)} chunks\n") for i, p in enumerate(pieces): print(f"--- chunk {i} ({len(p)} chars)") print(p, "\n") 

Output:

2 chunks --- chunk 0 (567 chars) Refunds are started by the merchant, either from the dashboard (Transactions, then Refund) or through the API. You can refund the full amount or part of it. Money returns to the customer's original payment method within 5 to 10 business days, depending on the customer's bank. Merchants can refund a transaction up to 90 days after it was made. The fees Brightlane charged on the original transaction are not returned when you refund. Refunds are taken from your Brightlane balance. If your balance is not enough, the refund is deducted from your next settlement. --- chunk 1 (297 chars) Customers who want a refund should contact the merchant they paid. Brightlane cannot issue a refund on behalf of a merchant. Failed transactions: if a customer was debited but the payment failed, the money is automatically reversed to the customer within 24 hours. Some banks take up to 72 hours. 

Two things are worth noticing. First, the splitter cuts between paragraphs, not in the middle of one. It tried blank lines first, and since the first five paragraphs fit within 600 characters, they stayed together. Second, the two chunks have a clear topic each: how a merchant refunds, and what a customer or a failed payment should do. I set chunk_overlap=50, but you can see there is no repeated text between the chunks. Overlap can only repeat whole paragraphs that are shorter than the overlap size, and every paragraph here is longer than 50 characters.

A splitter only cuts text, so it doesn’t know this came from the refunds page. Chunk 1 starts with “Customers who want a refund...”, which makes sense here, but a chunk such as “Fees are capped at ₦2,000” would leave a reader guessing whose fees. The fix is to attach the page heading to every chunk before moving on:

chunks = [f"Refunds\n\n{p}" for p in pieces] 

Now every chunk starts with Refunds, so it carries its topic wherever it ends up. On pages with similar content, like the Nigeria and Ghana pricing pages, that heading is what keeps their chunks apart.

Once you have decided your chunk strategy we can move to the next step which is embedding.

Embedding

Now that we have split our documents using RecursiveCharacterTextSplitter, we have a clean list of text chunks. However, a plain text search matches words, not meaning. If a customer asks "How long do payouts take?" and your chunk says, "Transfers settle within two business days", a traditional keyword search, might miss it because the exact words don't match.

Embeddings solve this. An embedding model reads a piece of text and turns it into a list of numbers that represents its meaning. Texts that mean similar things get similar numbers, whatever words they use. Searching by meaning then becomes a simple question: which stored chunks have numbers closest to the question's numbers?

What an embedding looks like

Think of every chunk getting an address on a map. Chunks about similar things live close together, and chunks about different things live far apart. When a customer asks a question, the question gets an address too, and we look at which chunks are nearest.
The model I used, bge-small-en-v1.5, gives every text 384 numbers. You never need to read them. What matters is the distance between them. "How long do payouts take?" and "Transfers settle within two business days" share almost no words, but the model has learned they are about the same thing, so their addresses are close.

Let’s embed three short chunks and one question, then see which chunk is closest in meaning to the question.

  1. Install the library, you can skip this step if you have already done it in the prerequisite
pip install sentence-transformers 
  1. Create a file called embed_demo.py and paste in this code:
from sentence_transformers import SentenceTransformer, util model = SentenceTransformer("BAAI/bge-small-en-v1.5", device="cpu") chunks = [ "Transfers settle within two business days.", "Fees are capped at ₦2,000 per transaction.", "Customers can raise a dispute up to 120 days after the transaction.", ] question = "How long do payouts take?" chunk_vectors = model.encode(chunks, normalize_embeddings=True) question_vector = model.encode( "Represent this sentence for searching relevant passages: " + question, normalize_embeddings=True, ) scores = util.cos_sim(question_vector, chunk_vectors)[0] for chunk, score in zip(chunks, scores): print(f"{score:.3f} {chunk}") 

Here is what the code does:

  • It loads the embedding model. The first run downloads it, so it takes a little longer. device="cpu" makes it run on your processor, which avoids errors on laptops with an unsupported GPU.
  • It turns each chunk into a vector of 384 numbers.
  • It turns the question into a vector the same way, with the prefix that BGE models expect on questions.
  • It scores how close the question is to each chunk, using cosine similarity, and prints the scores.
  1. Run it and read the output:
python3 embed_demo.py 
0.643 Transfers settle within two business days. 0.533 Fees are capped at ₦2,000 per transaction. 0.586 Customers can raise a dispute up to 120 days after the transaction. 

The "transfers" line ranks first, but the scores are close, which is normal for a small model. What matters is the ranking, not the absolute numbers.
Higher score means closer in meaning. Notice the best match shares no keyword with the question, which is exactly what a keyword search can't do.

Three things to remember

  • Use the same model for chunks and questions. Vectors from different models live on different maps, so comparing them is meaningless.
  • Some models want a prefix on questions. The bge models expect the line "Represent this sentence for searching relevant passages:" in front of the question, and nothing in front of the chunks. It's easy to miss, and results quietly get worse without it.
  • Changing the model means starting again. If you switch models later, every chunk has to be embedded again.

I used the bge-small local model becuase it is free and runs on a normal laptop CPU, and keeps your document on your own machine. For production, you may want a hosted embedding API such as OpenAI or Voyage AI. They need no servers of your own, but they cost money per call, and your text is sent to the provider.

Beyond embeddings

Embeddings alone are not enough. These additions matter just as much:

  • Keep metadata with every chunk, such as the source URL, country and last-updated date. It lets you hide outdated pages and shows users only what applies to them, becuase embeddings measure similarity, not correctness.
  • Combine keyword and vector search (hybrid search). Vector search captures overall meaning, while keyword search catches exact terms and specific details.
  • Test continuously. Keep a set of questions with known answers, rerun it after every change, and log real customer questions so you can see what your documents are missing.

Conclusion

Chunking and embedding are the foundation of every RAG system. Chunking decides what each searchable piece contains, and embedding turns those pieces, and the customer’s question, into numbers that capture meaning.

What's next

Start with the simplest setup that works. The next step is to store the chunks and vectors in a vector database and attach metadata like country, so you can filter and retrieve only what applies to each user.