In RAG pipelines, text chunking usually forces an uncomfortable trade-off:
- Fast window splitters (like a raw
tiktokenloop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers. - Structure-aware splitters try to respect document layout, but they are often 4×–11× slower, pull in dozens of dependencies, and frequently overshoot hard token limits (returning 513–514 tokens on a 512-token budget when headings or BPE boundary joins aren't re-counted). With text-chipper v1.1.0, the goal was to eliminate that trade-off: build the fastest chunker that guarantees both strict token budgets and Markdown/table structural awareness, backed by reproducible benchmarks. Here is what the benchmark numbers look like, the two major problems we solved in
v1.1.0(Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets. --- ## 1. The Benchmark: Speed + Strict Token Compliance + Structure We benchmarkedtext-chipperagainst seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict 512-token budget (cl100k_base,overlap=0).
### Throughput (Single-Core Geometric Mean across 12 Workloads)
-
text-chipper(default Markdown- and table-aware strict mode): 37.1 MiB/s - Plain
tiktokenwindow (no structure, cuts words/code): 15.5 MiB/s - Other chunking libraries: 3.2 to 9.5 MiB/s (
text-chipperis 3.9× to 11.6× faster) At the same time: - 0 chunks over 512 tokens across all 12 workloads (compared to 15 over-limit chunks in LangChain up to 514 tokens, and 18 each in
tiktokenwindow and Chonkie'sTokenChunker). - 100% of measured small code fences kept intact (392 in
markdown_docand 3,434 incode_heavy).
- 0 runtime dependencies (~4 MB installed wheel size, ~2 ms import time).
2. New in v1.1.0: Markdown Table-Aware Splitting (Without Breaking Source Offsets)
Consider a 200-row financial or inventory table in Markdown:
| Row | Item | Category | Price | | --- | --- | --- | ---: | | 001 | Product-1 | Hardware | $10 | ... | 200 | Product-200 | Hardware | $2000 | When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:
- A cut happens in the middle of a row (
| 045 | Prodin chunk 1,uct-45 | Hardware |in chunk 2). - Chunks 2..N lose the table header (
| Row | Item | Category | Price |), leaving the embedding model and LLM with floating rows of numbers that lack column names. ### Howtext-chipperv1.1.0 solves it—while keepingdoc[start:end] == chunk.textA core invariant oftext-chipper(verified across 26.9 million checks in our release test suite) is thatchunk.textis always an exact contiguous slice of the normalized input:
assert doc[chunk.start_index : chunk.end_index] == chunk.text If we mutated chunk.text on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.
Instead, text-chipper treats the repeated table header as structural context, just like section headings (chunk.headers):
- Tables that fit stay whole: If the entire GFM table fits within
max_length, it is never split. - Row-boundary splitting: If an oversized table must be split, row boundaries are promoted to top-level structural cut points (
R_ITEM), so rows are kept whole whenever they fit. - Chunk 1 of the table starts at the table's opening line in the source text, so
chunk.textalready begins with the verbatim header and| --- |delimiter row (chunk.table_headerisNone). - Chunks 2..N of the table keep
chunk.textas the exact contiguous slice of data rows, attach the header + delimiter lines aschunk.table_header, and automatically prepend them inchunk.contextualized_text. - Budget-enforced: The token cost of the repeated table header is reserved up front during row packing and re-verified on
chunk.contextualized_textby the strict final gate.
from text_chipper import TextChipper header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |" rows = [f"| {i:03d} | Product-{i} | Hardware | ${i * 10} |" for i in range(1, 201)] doc = "# Inventory\n\n" + header + "\n" + "\n".join(rows) + "\n" chipper = TextChipper(max_length=120, overlap=0, encoding="cl100k_base") chunks = chipper.chunk(doc) for idx, c in enumerate(chunks): assert c.token_count <= 120 assert chipper.count_tokens(c.contextualized_text) == c.token_count assert header in c.contextualized_text # Every chunk has the table header! assert doc[c.start_index : c.end_index] == c.text # And exact source offsets still hold! 3. New in v1.1.0: Zero-GIL Hugging Face Tokenizers & Multi-Tokenizer Guards
If you run local embedding models (BAAI/bge-large-en-v1.5, nomic-embed-text, jina-embeddings) or local LLMs (Qwen, Llama-3, Mistral), counting tokens with OpenAI's cl100k_base is dangerous: a 500-token chunk under cl100k_base can easily be 610+ tokens under your model's actual tokenizer.
Even worse, if you pass a Python AutoTokenizer callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.
Native Rust handoff for AutoTokenizer and from_pretrained
In text-chipper v1.1.0:
- You can call
TextChipper.from_pretrained("BAAI/bge-large-en-v1.5", max_length=510)directly (resolvingtokenizer.jsonfrom a local folder/file or viahuggingface_hub/tokenizers/transformers). - Or you can pass an already-loaded
transformers.PreTrainedTokenizerFast(AutoTokenizer.from_pretrained(...)) ortokenizers.Tokenizerviatokenizer=. - At initialization,
text-chipperextracts the serialized JSON from.backend_tokenizer.to_str()and loads it directly into Hugging Face's native Rusttokenizerscrate (Backend::from_tokenizer_bytes). During chunking, zero Python calls are made and the GIL is completely released. - Need a custom Python counter? You can also pass any
callable(text) -> intor object with.encode(text). ### Multi-Tokenizer Guards (guard_tokenizers) What if your pipeline sends the same chunk to both an embedder (e.g.bge-large-en-v1.5) and a cross-encoder reranker (bge-reranker-v2-m3) or LLM with a different vocabulary?
from transformers import AutoTokenizer from text_chipper import TextChipper embed_tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5") rerank_tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3") chipper = TextChipper( max_length=510, # leave 2 tokens for [CLS] and [SEP] under a 512 limit overlap=32, tokenizer=embed_tok, guard_tokenizers=[rerank_tok], ) for chunk in chipper.chunk(markdown_text): # Guaranteed <= 510 tokens under BOTH BGE and BGE-reranker-v2-m3! print(chunk.token_count, chunk.contextualized_text) 4. Bounded-Memory Streaming on 500+ MiB Files
When processing massive datasets or log exports, loading a 514.3 MiB file into a single Python string and calling chunk() spikes process memory by 4.6 GB.
Because text-chipper's BlockParser and SmartChunker are incremental line-driven state machines, from_file_stream(path) streams the 514.3 MiB file in 25 seconds (20.5 MiB/s on a single core) with only 56 MB of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.
Try It Out
text-chipper v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:
pip install -U text-chipper - GitHub Repository & Benchmarks: https://github.com/kameh0use/text-chipper
- PyPI: https://pypi.org/project/text-chipper/
- Rust Crate (
text_chipper): https://crates.io/crates/text_chipper If you hit any strange Markdown tables or tokenizer edge cases in your RAG corpus, open an issue—I'd love to add them to the benchmark and fuzz suite!



