Hi everyone, I've published text_chipper, a Rust crate (with Python bindings via maturin) for splitting large text/Markdown files into token-budgeted chunks for RAG pipelines.
Design highlights:
- Strict token budgets: Every finalized chunk is re-tokenized in the exact form it is returned (including heading-path prefixes), guaranteeing
token_count <= max_length. If a single grapheme cluster exceeds the budget, it returns an error instead of emitting an oversized chunk. - Grapheme-safe & offset-preserving: Splits never break Unicode grapheme clusters, and output offsets map cleanly back to the original input.
- Lightweight Markdown block parser: Tracks heading paths and keeps fenced code blocks and list items intact when they fit.
- Bounded-memory streaming: On a 514.3 MiB file,
from_file_stream()grew memory by ~56 MB (vs ~4.6 GB when loading the whole file into a Python string).
Links:
- Crates.io: crates.io: Rust Package Registry
- GitHub: GitHub - kameh0use/text-chipper: High-performance, Markdown-aware, strictly budget-enforced text chunking for RAG and LLM pipelines. Built in Rust with Python bindings. · GitHub
- Benchmark write-up: 513 Tokens in a 512-Token Chunk: Why I Recount Every Chunk - DEV Community
I'd really appreciate feedback from Rust folks on:
- API design for custom token counters (currently Rust-only)
- Whether the strict-mode recount tradeoff looks reasonable
- Anything un-idiomatic in the crate
1 post - 1 participant

