How Does a Text Splitter Actually Chunk Your Documents?

The previous post in this series — What Is RAG, and Why Does Your LLM Need It? — walked the whole pipeline: extract the text, chunk it, embed each chunk, store the vectors. This post goes deep in the second step; chunking.

A document goes in — a PDF, a wiki page, a support ticket — and what comes out isn't the document. It's a few hundred short passages, each just a few sentences long. Those passages are chunks, and producing them is chunking. It happens once, up front, before the system has been asked anything.

Those chunks are what gets searched, and they're the only part of your documents the model ever sees. So where the cuts land decides what can be found later. Which raises the two questions this post answers: what makes a cut a good one, and what actually decides where it lands?

Here's the route. Nothing to work out from it now — it'll make sense by the end.

We'll get there in four steps. First, the four requirements a good chunk has to satisfy, and why they pull against each other. Then two obvious strategies, each of which solves half the problem and ignores the other half. Then the two-step design — SPLIT and PACK — that satisfies all four at once. And finally what that design means for the two settings you actually configure, chunk_size and chunk_overlap, where the second one turns out not to mean what its name suggests.

Why chunking exists, briefly

Two hard constraints force a document to be cut up before it's searchable: embedding models accept only so much text at once, and retrieval works better on small, focused chunks than on large, unfocused ones.

Those turn out to be one problem. An embedding is a fixed-size vector, no matter how much text goes in — one paragraph or ten pages, same number of dimensions back. A vector built from ten pages covering four topics lands near the blurry average of all four, not close to any one of them.

One vector can only mean one thing. So a chunk has to be about one thing.

There's a second constraint: retrieval only ever hands back a whole chunk, never something assembled from several. Wherever the cuts land decides, once, what this system can even retrieve later.

That's the short version — enough to follow everything below. This post makes the full case, including the context window and what happens downstream once the chunks exist.

What a good chunk has to be

Four requirements apply at once, and they don't all pull the same way.

Requirements 1 and 2 pull toward smaller chunks. Requirements 3 and 4 pull toward bigger ones. There's no setting that maximizes all four at once — every chunking strategy is a position on that trade-off, and everything below is measured against this table.

The next few sections use a size limit of 100 characters — small enough to do the arithmetic in your head. Later sections scale it up to real document sizes; the arithmetic works the same way.

Attempt #1: cut every 100 characters

Count, and cut on a fixed size. The obvious first move: guarantee size by counting characters.

chunks = [text[i : i + 100] for i in range(0, len(text), 100)] 

Run it on the running example for this post —

text = ( "Django models map to tables.\n\n" "Each field is a column.\n\n" "Migrations are versioned Python files that describe schema changes and " "apply them to the database in a deterministic order every single time." ) 

— and here is the chunk boundary, the place where one chunk ends and the next begins:

[ 0] '...Migrations are versioned Python files that de' [100] 'scribe schema changes and apply them to the database...' 

describe is now de in one chunk and scribe in the other. The counter has no idea a word is even there, let alone an idea.

Boundaries are worth naming because the rest of this post keeps coming back to them. Cut a document into n chunks and you create n − 1 boundaries, and every one of them is a place where an idea might have been severed. Whether the two chunks either side of a boundary share any text — and how much — is the subject of the last half of this post.

Size is guaranteed. Meaning is not.

The diagnosis is simple: the cut position was chosen by a counter, not by the text. Character 100 has no relationship whatsoever to where a word, a sentence or an idea ends.

Attempt #2: cut only where the text already breaks

The fix looks obvious: stop counting, and cut at the text's own break points instead.

(2a) By Space

Split on spaces (text.split(" ")) and 'to', 'a', 'is' come out as standalone chunks — safe cut points, but useless size, broken meaning.

(2b) By Paragraph

Split on paragraphs (text.split("\n\n")) and the running example's three paragraphs come out [28, 23, 141] characters long, against a 100-character limit — the third is over by 41%, and there's no lever to pull. Real documents may have 20-character paragraphs and 3,000-character ones sitting side by side.

Where that leaves us

Counting solves size and ignores meaning. Respecting structure solves meaning and ignores size. Score them against the four requirements and the pattern is exact:

Read the columns, not the rows: every requirement has a ✅ somewhere, held either by the methods that count or by the method that respects the text's own structure. No method holds all four, because counting and respecting structure are two different kinds of work, and each method here only does one of them.

Nobody has failed at chunking. We've been trying to do two different jobs with one mechanism.

Two jobs, two mechanisms

Look back at that table. Every ✅ in it comes from one of two kinds of work: counting, which guarantees the size limit, or respecting the text's own structure, which guarantees the other three. No method does both, because a pass that's watching a running total isn't, at the same time, hunting for a paragraph break — one pass, one job.

So do them one after another, and let each do only what it's good at. First, go through the text once and mark every place it would be safe to cut — a paragraph end, a sentence end, a space — without asking how big anything is. Then go through what that produced and gather it into groups that fit under the size limit, without asking what any of it means. The second step gets to ignore meaning only because the first step already threw out every cut that would have damaged it.

Those two steps need names, and there aren't official ones to reach for — so this post coins its own: call the first step SPLIT and the second PACK. Treat them as labels for this explanation, not as terminology you'll find anywhere else — LangChain's own methods, which both turn up later, are called _split_text and _merge_splits. SPLIT is the step that respects the text's own structure; PACK is the step that counts.

SPLIT answers one question: where am I allowed to cut? A blank line is a legal cut point — a place where cutting wouldn't damage anything — and so is the space between two words. The middle of the word describe is not: that's the cut attempt #1 made. Size never enters into it.

PACK answers the other question: how many of those go into one chunk? It glues what SPLIT produced back together, in order, until a chunk is nearly full, then starts another. Size is the only thing it looks at.

Watch both run on the running example.

SPLIT starts the same way attempt 2b did: cut on the blank lines. Three paragraphs come out, [28, 23, 141] characters. Each one is a piece — the word for a single unit of SPLIT's output.

A piece is scratch work, not a deliverable: it never reaches you, and it never reaches your vector database. It only ever goes to PACK.

The 28- and 23-character pieces are already comfortably under the 100-character limit, so SPLIT leaves them exactly as they are. The 141-character one is over, so SPLIT isn't finished with it — it goes back to work on that paragraph alone, hunting for a finer cut point inside it. There's no blank line in there, so it drops down a step, then another, and ends up cutting the paragraph at spaces: 22 individual words, each one safely under the limit.

Now PACK takes over. Its rule is simple: walk the pieces in order, keep a running total, and the moment the next piece would push that total past 100, close the current chunk and start a new one. Give it the 28- and 23-character pieces and they fit together into a single chunk, comfortably under the limit.

Give it the 22 words and they fill chunk after chunk, each one closing just before it would cross 100. What PACK hands back is a chunk — this is what actually gets embedded, stored, and handed back later by retrieval.

Neither step had to compromise. SPLIT never broke a word to hit a size, because size was never its problem. PACK never had to worry about breaking a word, because SPLIT had already made that impossible before PACK saw a single piece.

This is exactly what LangChain's RecursiveCharacterTextSplitter does:

from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) chunks = splitter.split_text(text) 

The name is a mouthful, but it's just a description of the machine you've been building:

• Splitter — it cuts text into pieces. That's SPLIT.
• Character — it measures length by counting characters, not words and not tokens.
• Recursive — when a piece comes out too big, it runs the same procedure again on that piece alone, cutting at something finer. That's exactly what just happened to the 141-character paragraph above — the next section walks through it step by step.

The two settings in that code are chunk_size=100, the same limit used throughout, and chunk_overlap=20, which does nothing at all until step 5 of the walkthrough below. SPLIT has never heard of it.

Inside the splitter, step by step

Everything up to here has been the design. Now watch the real machine run it, one move at a time, on the same three paragraphs and the same 100-character limit.

Start with the list SPLIT works from. SPLIT doesn't figure out good cut points by reading the text — it's handed a list of separators, written best-first:

• "\n\n" — a blank line. A paragraph break is the clearest cut point a document has.
• "\n" — a single line break. Weaker, but still a real cut point.
• " " — a space between two words. Weak, though it never breaks a word in half.
• "" — the empty string, which means "cut between any two characters". The last resort.

That list is the separator ladder: ["\n\n", "\n", " ", ""]. It isn't a new idea — it's the priority order SPLIT already needed, written down. SPLIT starts on the top rung and moves down only when it has to. (LangChain's method for this is _split_text(text, separators).)

The bottom rung is the one to remember. "" is the empty string, so every text contains it, always. That's what keeps the ladder from ever running out of options: there is always somewhere left to cut. A later section takes that rung away, and watches chunk_size stop being a guarantee.

SPLIT's rule has three parts:

• Pick a rung. Find the highest one that actually appears in this text, and cut the text there.
• Check each piece. Under chunk_size? Then that piece is finished, and it goes to PACK.
• Re-run on anything still too big. Take that piece on its own and apply these same three rules to it, starting one rung lower.

That last part is the whole meaning of "recursive": the procedure calling itself on part of its own output. Because each self-call starts a rung lower, every trip through cuts finer than the last.

The walkthrough below runs in five steps. The first three are SPLIT cutting the text down into pieces. The last two are PACK building chunks out of those pieces. There's a diagram of all five together at the end of the walkthrough, once each step means something.

Step 1 — cut on the blank lines

The text has blank lines in it, so the top rung, "\n\n", is the separator SPLIT uses. Cutting there gives three pieces:

[0] len= 28 'Django models map to tables.' [1] len= 25 '\n\nEach field is a column.' [2] len=143 '\n\nMigrations are versioned Python files that describe schema changes and apply them to the database in a deterministic order every single time.' 

Two things to notice, and the second one matters more.

The lengths shifted. Earlier, these same three paragraphs measured [28, 23, 141]; here they're 28, 25 and 143. The text didn't change. The splitter keeps each separator attached to the front of the piece that follows it instead of throwing it away, so pieces 1 and 2 each carry a \n\n — two extra characters apiece. That whitespace gets stripped later, from the outer edges of a finished chunk, which is why the finished chunks start at Migrations rather than at a blank line.

PACK runs now, not at the end. Pieces 0 and 1 are both under 100, so both are finished. Piece 2 is 143 and isn't. The moment SPLIT sees that third piece is too big, it hands the finished pair to PACK — and PACK closes them into a chunk right there, before SPLIT has started cutting the third piece up.

That chunk is done and gone. It's the first of the three chunks this document produces: 53 characters, the two short paragraphs and nothing else. It never picks up a single word from the third paragraph, because it was sealed before the third paragraph had been cut up at all.

That ordering is the one place the design and the code differ. "Two jobs, two mechanisms" described SPLIT and PACK as two clean passes: mark every legal cut in the whole document, then glue the results into chunks. As a picture of the design that's exactly right, and it's the picture to keep. The code just doesn't run in that order. PACK is called from inside SPLIT, so it runs many times over one document — once per group it's handed — and never once over everything SPLIT eventually produces. The two steps interleave.

Step 2 — the piece that didn't fit

Piece 2 is over the limit, so SPLIT recurses into it alone, carrying the rest of the ladder: ["\n", " ", ""]. The top rung of what's left is "\n", so that's the cut:

[0] len= 1 '\n' [1] len=142 '\nMigrations are versioned Python files that describe schema changes and apply them to the database in a deterministic order every single time.' 

Barely any progress — piece [1] is still 142 characters, over the limit again. Down another rung.

Step 3 — down to single words

Splitting that piece on " " gives 22 pieces, one per word.

[0] len= 11 '\nMigrations' [1] len= 4 ' are' [2] len= 10 ' versioned' [3] len= 7 ' Python' [4] len= 6 ' files' [5] len= 5 ' that' [6] len= 9 ' describe' [7] len= 7 ' schema' [8] len= 8 ' changes' [9] len= 4 ' and' [10] len= 6 ' apply' [11] len= 5 ' them' [12] len= 3 ' to' [13] len= 4 ' the' [14] len= 9 ' database' [15] len= 3 ' in' [16] len= 2 ' a' [17] len= 14 ' deterministic' [18] len= 6 ' order' [19] len= 6 ' every' [20] len= 7 ' single' [21] len= 6 ' time.' 

All 22 are under 100, so the recursion stops here — nothing has to fall to the bottom rung and get cut character by character.

SPLIT is finished. Everything it was ever going to produce now exists: the two short paragraphs from step 1, which are already packed and gone, and these 22 words. So there's exactly one job left — turn 22 words into chunks. That's PACK, and it's the next two steps.

Step 4 — PACK fills a chunk

PACK has almost no memory. It keeps two things, and that's the whole of it:

• a buffer — the pieces going into the chunk it's building right now
• a total — their combined length

Its rule is just as short. Take the pieces in order, add each one to the buffer, and keep the total updated. When the next piece would take the total over chunk_size, close the chunk and start a new one.

Here it is on the 22 words. They go in one at a time, and the total climbs:

'\nMigrations' 11 -> total 11 ' are' 4 -> total 15 ' versioned' 10 -> total 25 ' Python' 7 -> total 32 ' files' 6 -> total 38 ' that' 5 -> total 43 ' describe' 9 -> total 52 ' schema' 7 -> total 59 ' changes' 8 -> total 67 ' and' 4 -> total 71 ' apply' 6 -> total 77 ' them' 5 -> total 82 ' to' 3 -> total 85 ' the' 4 -> total 89 ' database' 9 -> total 98 

First fifteen words in, 98 characters. The sixteenth word is ' in', three characters long. 98 + 3 = 101, cross the limit — so the chunk closes.

Two details here, both worth slowing down for.

' in' is not in the chunk. The chunk closes with what was already in the buffer. The word that triggered the close is exactly the word being kept out — it goes on to start the next chunk instead.

The chunk is 97 characters, not 100. The buffer totalled 98, and the leading \n gets stripped off the finished chunk, leaving 97. PACK stopped there because the only way to get closer to 100 was to take a word that didn't fit, and it moves whole words only — never parts of them. That restriction is the entire reason chunks don't end mid-word, and the gap under the limit is what it costs. chunk_size is a ceiling, not a target.

That's the second of the three chunks, finished and emitted. Seven of the 22 words are still outside it: ' in', which caused the close, and the six that come after it.

Step 5 — the drain, and the words that survive

Now the part the fill rule left out. The chunk has been emitted, but the buffer still holds all fifteen words that went into it.

Before the trace, one thing worth fixing in place, because it's the easiest thing here to get backwards. The drain is not cleanup. It isn't there to empty the buffer. It exists to decide exactly one thing: what the next chunk in this run starts with. Whatever it drops, it drops because it isn't needed for that.

Which tells you when it runs — only when a chunk closes and another one is still coming in the same run. That's what "a piece didn't fit" amounts to: ' in' couldn't go in, so a chunk had to close and a new one has to start, and something has to decide what carries over into it. In the code, the pop loop sits inside that very if.

It also answers the obvious follow-up: if the drain is skipped, who empties the buffer? Nobody does. PACK's buffer is a local variable belonging to one run — created empty when the run starts, gone when the run returns. There is no line anywhere in the splitter that clears it. That is simply what became of the buffer holding those two short paragraphs in step 1: PACK ran out of pieces, emitted what it had, and returned, and the buffer went with it.

Here, though, PACK does have a next chunk to seed. So it drops words from the front, one at a time, for as long as what's left is bigger than chunk_overlap:

pop '\nMigrations' -> total 87 87 > 20, keep going pop ' are' -> total 83 pop ' versioned' -> total 73 pop ' Python' -> total 66 pop ' files' -> total 60 pop ' that' -> total 55 pop ' describe' -> total 46 pop ' schema' -> total 39 pop ' changes' -> total 31 pop ' and' -> total 27 pop ' apply' -> total 21 21 > 20, one more pop ' them' -> total 16 16 is not > 20 -> stop 

Three words are left standing: ' to', ' the', ' database' — 16 characters.

Those three words are the overlap. Nothing was copied to produce them. They're baked into the chunk that just closed, which can't change now; and they're also still sitting in the buffer, which is where the next chunk gets built. The same three words end up in two chunks at once.

Why drop from the front? Because the words at the back of the buffer are the ones touching the boundary — where the chunk that just closed meets the one about to start — and they're the context the next chunk needs. The words at the front are furthest from the next chunk, and the chunk that just closed already covers them.

With the drain done, PACK simply continues on filling, from where it left, ' in'. ' in' finally goes in, bringing the total to 19, the last six words follow, and the words run out at a total of 60. There's nothing left to overflow anything, so the buffer is emitted as the third and final chunk — 59 characters after stripping.

That's the whole machine. Which leaves one sentence, and it explains every strange number in the rest of this post:

Overlap is a leftover, not an allocation. Nobody measures out 20 characters. 20 is where the dropping stops, and the overlap is whatever happens to still be standing.

Because PACK only ever moves whole pieces, that leftover lands on whatever number the piece sizes allow — 16 here, and 15 once the leading space is stripped. Not 20. chunk_overlap is a ceiling too.

Here are all five steps in one picture, in the order they ran:

All five steps, in code

Two functions. Steps 1 to 3 are _split_text; steps 4 and 5 are _merge_splits. Both are simplified for readability — shorter names, no class, no generics — but the control flow is the real one. I checked the _merge_splits below against the library by capturing every call the real splitter made across four documents and six chunk_size/chunk_overlap settings, and running this version on the same inputs: 168 calls, identical output every time.

Steps 1 to 3 — SPLIT:

def _split_text(text, separators): # use the first separator this text actually contains;  # "" means "split into individual characters"  separator, finer_separators = _first_present(text, separators) pieces = text.split(separator) if separator else list(text) good, final_chunks = [], [] for piece in pieces: if len(piece) < chunk_size: # strictly less than  good.append(piece) else: if good: final_chunks += _merge_splits(good, separator) good = [] if finer_separators: final_chunks += _split_text(piece, finer_separators) # recurse  else: final_chunks.append(piece) # nothing finer left; keep it oversized  if good: final_chunks += _merge_splits(good, separator) return final_chunks 

Both _merge_splits(...) calls in there are PACK, and the walkthrough used one of each. The first fired in step 1, the moment SPLIT met the oversized third paragraph — that's the flush that produced chunk [0]. The second fired at the end of the loop that had just cut the 22 words, with nothing left to walk; that's the run you followed through steps 4 and 5. PACK never runs anywhere except from inside SPLIT, which is the interleaving, sitting in the code itself.

Steps 4 and 5 — PACK. This is the function those two calls land in. The drain is the while loop in the middle:

def _merge_splits(pieces, separator): chunks = [] buffer, total = [], 0 # step 4: the buffer, and its running total  def close(): # join, strip, and drop an all-blank result  text = separator.join(buffer).strip() if text: chunks.append(text) for piece in pieces: added = len(piece) + (len(separator) if buffer else 0) if total + added > chunk_size: # this piece does not fit  if total > chunk_size: warn(f"Created a chunk of size {total}, longer than {chunk_size}") if buffer: close() # close, WITHOUT this piece  # step 5 — the drain. Nested this deep on purpose: it runs only  # when a piece did not fit, which is the only time another chunk  # is coming that could use whatever survives.  while total > chunk_overlap or ( total + added > chunk_size and total > 0 ): total -= len(buffer[0]) + (len(separator) if len(buffer) > 1 else 0) buffer.pop(0) # pop from the FRONT  # whatever is still in `buffer` becomes the next chunk's head  buffer.append(piece) total += added close() # pieces ran out — no drain runs here  return chunks # `buffer` was local; it ends with the call 

Look at how deeply that while is nested: inside if total + added > chunk_size — a piece didn't fit — and then inside if buffer — there's something to close. That nesting is the whole answer to why chunk [0] has no overlap. Its run was handed two pieces totalling 53, so the outer if never fired even once, and execution never got within two levels of the drain. It fell out of the loop, hit the if buffer at the bottom, emitted one chunk and returned.

Notice too what the function doesn't contain: any line that empties buffer. It's created empty on the way in, and it's gone when the function returns. Compare good = [] up in _split_text — SPLIT clears its pending list deliberately, because SPLIT carries on walking pieces after a hand-off. PACK has nothing to carry on for.

What comes out

Both halves have now run. Here is everything the splitter hands back:

chunk [0] len= 53 'Django models map to tables.\n\nEach field is a column.' chunk [1] len= 97 'Migrations are versioned Python files that describe schema changes and apply them to the database' chunk [2] len= 59 'to the database in a deterministic order every single time.' 

Chunk [0] was closed back in step 1. Chunks [1] and [2] are the two that steps 4 and 5 built out of the 22 words.

Now look closely at where chunk [1] ends and chunk [2] begins:

[1] ... schema changes and apply them to the database [2] to the database in a deterministic order ... 

to the database — those are the three words that survived the drain in step 5, 15 characters once the leading space is gone. They're in both chunks. That shared text is overlap, and the previous post covers why chunks are built to share their edges at all.

So one of the two boundaries got a bridge, and the other got nothing:

Same document, same run, the same chunk_overlap=20 for both. So why did one boundary get 15 characters and the other none at all?

Why the first boundary has no overlap

There are two separate reasons, and either one alone would have been enough.

Reason 1 — nothing ever failed to fit in that buffer. Chunk [0] came from the PACK run that was handed exactly two pieces, 28 and 25 characters. They went into the buffer exactly the way the 22 words did in step 4 — same buffer, same rule, running total 28 then 53. The difference is how that run ended: it ran out of pieces. Nothing overflowed, so the branch the drain lives in never fired, and the buffer was joined and returned untouched.

What actually closed that chunk was SPLIT meeting the oversized third paragraph and handing off what it had, back in step 1. That's a different kind of close, and it carries nothing forward.

Reason 2 — there was nowhere to carry it to. Even if the drain had run, its survivors would have gone nowhere. Chunk [0] came out of one PACK run; chunk [1] came out of a different one, on a completely different group of pieces. The buffer belongs to a single run — created empty, and gone when that run returns — so nothing from the earlier run can reach the later one.

That gives a rule worth keeping:

Overlap can only ever appear between two chunks emitted by the same PACK run. Every boundary where SPLIT switched groups is a hard cut — at any settings, on any document.

Seam [1]→[2] sits inside one run, on word-sized pieces, so the drain worked normally. Seam [0]→[1] spans two runs, so no mechanism existed that could have bridged it.

Three ways a chunk closes

Worth collecting, because only one of the three can produce overlap at all:

The rule underneath: popping happens when a chunk closes and the run keeps going. If the chunk closed because the run is ending — the pieces ran out, or SPLIT is leaving to go split something else — there's no next chunk to hand a tail to, so nothing is kept.

And even the top row can give you zero. The drain has a second condition alongside the one you watched. It also keeps popping while what's left, plus the piece that's about to go in, would still overflow.

That's a safety net. There's no point keeping a tail for the next chunk if the piece starting that chunk already needs the whole 100 characters to itself.

Which means the piece after a boundary helps decide whether that boundary gets a bridge at all. Here are two runs that are identical until the very last piece:

buffer when the chunk closes: [32, 27, 17, 10, 9] = 95 if next piece is 55 pop 32, 27, 17 -> [10, 9] = 19 19 + 55 = 74, fits -> stop. overlap 17 if next piece is 97 pop 32, 27, 17 -> [10, 9] = 19 19 + 97 = 116, no -> keep popping pop 10 -> [9] = 9 9 + 97 = 106, no -> keep popping pop 9 -> [] = 0 -> stop. overlap 0 

The 10 and the 9 are small enough to survive — 19 is under the 20 ceiling, and in the first run they do. The second run throws them out anyway, because nothing can sit beside a 95-character piece in a 100-character chunk.

There's one more way to get zero from that top row, and it's the most common of all in real documents. That's the next section.

The overlap you asked for is a ceiling

If you've ever written a sliding window by hand, overlap was exact: advance by 80, take a window of 100, and the last 20 characters repeat every single time. That's the intuition the name chunk_overlap inherits. PACK doesn't slide a window — it drains a buffe...