In RAG pipelines, text chunking usually forces an uncomfortable trade-off:

  1. Fast window splitters (like a raw tiktoken loop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers.
  2. Structure-aware splitters try to respect document layout, but they are often 4×–11× slower, pull in dozens of dependencies, and frequently overshoot hard token limits (returning 513–514 tokens on a 512-token budget when headings or BPE boundary joins aren't re-counted). With text-chipper v1.1.0, the goal was to eliminate that trade-off: build the fastest chunker that guarantees both strict token budgets and Markdown/table structural awareness, backed by reproducible benchmarks. Here is what the benchmark numbers look like, the two major problems we solved in v1.1.0 (Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets. --- ## 1. The Benchmark: Speed + Strict Token Compliance + Structure We benchmarked text-chipper against seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict 512-token budget (cl100k_base, overlap=0). Scorecard: token limit, words, source text and code fences per library, with speed ### Throughput (Single-Core Geometric Mean across 12 Workloads) Bar chart: geometric-mean throughput of each library over all 12 workloads
  3. text-chipper (default Markdown- and table-aware strict mode): 37.1 MiB/s
  4. Plain tiktoken window (no structure, cuts words/code): 15.5 MiB/s
  5. Other chunking libraries: 3.2 to 9.5 MiB/s (text-chipper is 3.9× to 11.6× faster) At the same time:
  6. 0 chunks over 512 tokens across all 12 workloads (compared to 15 over-limit chunks in LangChain up to 514 tokens, and 18 each in tiktoken window and Chonkie's TokenChunker).
  7. 100% of measured small code fences kept intact (392 in markdown_doc and 3,434 in code_heavy).

- 0 runtime dependencies (~4 MB installed wheel size, ~2 ms import time).

2. New in v1.1.0: Markdown Table-Aware Splitting (Without Breaking Source Offsets)

Consider a 200-row financial or inventory table in Markdown:

| Row | Item | Category | Price | | --- | --- | --- | ---: | | 001 | Product-1 | Hardware | $10 | ... | 200 | Product-200 | Hardware | $2000 | 

When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:

  1. A cut happens in the middle of a row (| 045 | Prod in chunk 1, uct-45 | Hardware | in chunk 2).
  2. Chunks 2..N lose the table header (| Row | Item | Category | Price |), leaving the embedding model and LLM with floating rows of numbers that lack column names. ### How text-chipper v1.1.0 solves it—while keeping doc[start:end] == chunk.text A core invariant of text-chipper (verified across 26.9 million checks in our release test suite) is that chunk.text is always an exact contiguous slice of the normalized input:
assert doc[chunk.start_index : chunk.end_index] == chunk.text 

If we mutated chunk.text on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.
Instead, text-chipper treats the repeated table header as structural context, just like section headings (chunk.headers):

  1. Tables that fit stay whole: If the entire GFM table fits within max_length, it is never split.
  2. Row-boundary splitting: If an oversized table must be split, row boundaries are promoted to top-level structural cut points (R_ITEM), so rows are kept whole whenever they fit.
  3. Chunk 1 of the table starts at the table's opening line in the source text, so chunk.text already begins with the verbatim header and | --- | delimiter row (chunk.table_header is None).
  4. Chunks 2..N of the table keep chunk.text as the exact contiguous slice of data rows, attach the header + delimiter lines as chunk.table_header, and automatically prepend them in chunk.contextualized_text.
  5. Budget-enforced: The token cost of the repeated table header is reserved up front during row packing and re-verified on chunk.contextualized_text by the strict final gate.
from text_chipper import TextChipper header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |" rows = [f"| {i:03d} | Product-{i} | Hardware | ${i * 10} |" for i in range(1, 201)] doc = "# Inventory\n\n" + header + "\n" + "\n".join(rows) + "\n" chipper = TextChipper(max_length=120, overlap=0, encoding="cl100k_base") chunks = chipper.chunk(doc) for idx, c in enumerate(chunks): assert c.token_count <= 120 assert chipper.count_tokens(c.contextualized_text) == c.token_count assert header in c.contextualized_text # Every chunk has the table header! assert doc[c.start_index : c.end_index] == c.text # And exact source offsets still hold! 

3. New in v1.1.0: Zero-GIL Hugging Face Tokenizers & Multi-Tokenizer Guards

If you run local embedding models (BAAI/bge-large-en-v1.5, nomic-embed-text, jina-embeddings) or local LLMs (Qwen, Llama-3, Mistral), counting tokens with OpenAI's cl100k_base is dangerous: a 500-token chunk under cl100k_base can easily be 610+ tokens under your model's actual tokenizer.
Even worse, if you pass a Python AutoTokenizer callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.

Native Rust handoff for AutoTokenizer and from_pretrained

In text-chipper v1.1.0:

  • You can call TextChipper.from_pretrained("BAAI/bge-large-en-v1.5", max_length=510) directly (resolving tokenizer.json from a local folder/file or via huggingface_hub / tokenizers / transformers).
  • Or you can pass an already-loaded transformers.PreTrainedTokenizerFast (AutoTokenizer.from_pretrained(...)) or tokenizers.Tokenizer via tokenizer=.
  • At initialization, text-chipper extracts the serialized JSON from .backend_tokenizer.to_str() and loads it directly into Hugging Face's native Rust tokenizers crate (Backend::from_tokenizer_bytes). During chunking, zero Python calls are made and the GIL is completely released.
  • Need a custom Python counter? You can also pass any callable(text) -> int or object with .encode(text). ### Multi-Tokenizer Guards (guard_tokenizers) What if your pipeline sends the same chunk to both an embedder (e.g. bge-large-en-v1.5) and a cross-encoder reranker (bge-reranker-v2-m3) or LLM with a different vocabulary?
from transformers import AutoTokenizer from text_chipper import TextChipper embed_tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5") rerank_tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3") chipper = TextChipper( max_length=510, # leave 2 tokens for [CLS] and [SEP] under a 512 limit overlap=32, tokenizer=embed_tok, guard_tokenizers=[rerank_tok], ) for chunk in chipper.chunk(markdown_text): # Guaranteed <= 510 tokens under BOTH BGE and BGE-reranker-v2-m3! print(chunk.token_count, chunk.contextualized_text) 

4. Bounded-Memory Streaming on 500+ MiB Files

When processing massive datasets or log exports, loading a 514.3 MiB file into a single Python string and calling chunk() spikes process memory by 4.6 GB.
Because text-chipper's BlockParser and SmartChunker are incremental line-driven state machines, from_file_stream(path) streams the 514.3 MiB file in 25 seconds (20.5 MiB/s on a single core) with only 56 MB of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.

Bar chart: memory growth of each library on a 514 MiB file, single-core lane

Try It Out

text-chipper v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:

pip install -U text-chipper