How we chunk long-form content and maintain translation quality with Claude API When we launched LectuLibre, our AI-powered book translation platform, we thought the hard part would be fine-tuning translation quality. It turned out the real engineering challenge was more mundane: Claude's token limits. A 300-page novel contains roughly 120,000 tokens. Claude 3 Sonnet has a 200K context window, so you might assume you can just send the whole book and ask for a translation. But the output token limit is only 4,096 tokens (8,192 for some models). Even if the input fits, asking for a 120,000-token translation in one call will hit a wall. The API will return a truncated response or an error. In this post, I'll walk through the chunking and orchestration system we built to translate long-form content with Claude reliably and cost-effectively, without losing narrative consistency. The core problem: input fits, output doesn't Books are naturally long. A typical 300-page EPUB produces around 100,000–150,000 tokens. Claude's context window can hold that input, but the model's maximum output tokens are capped (4,096 for Sonnet). You cannot ask the model to output a full book translation in one API call. Even if you could, quality suffers: the model may lose focus, repeat content, or hallucinate details over very long generations. We needed a system that would: Split the source text into manageable chunks that fit both input and output limits. Preserve context across chunks to keep terminology and style consistent. Process chunks in parallel to keep user wait times reasonable. Handle PDF extraction artifacts (headers, footers, page numbers) and EPUB chapter structures. Chunking strategy: sentence-aware, overlapping windows Our first naive attempt used textwrap or simple character-based slicing. That produced chunks ending mid-sentence, which led to awkward translations and lost pronouns. We quickly moved to a paragraph- and sentence-aware splitter. We use tiktoken with the cl100k_base encoding as a fast approximation for Claude's token count. It's not exact, but close enough for chunk sizing. The actual anthropic SDK also provides a count_tokens method if you need precision. Here's the chunker we use: import tiktoken import re enc = tiktoken.get_encoding('cl100k_base') def count_tokens(text: str) -> int: return len(enc.encode(text)) def split_into_chunks(text: str, max_tokens=8000, overlap_tokens=200) -> list[str]: paragraphs = re.split(r'\n\s*\n', text) chunks = [] current = '' current_tokens = 0 for para in paragraphs: para_tokens = count_tokens(para) if current_tokens + para_tokens > max_tokens and current: chunks.append(current) # keep last overlap_tokens worth from current as overlap overlap_text = current[-overlap_tokens:] if len(current) > overlap_tokens else current current = overlap_text current_tokens = count_tokens(current) current += '\n\n' + para current_tokens = count_tokens(current) if current: chunks.append(current) return chunks We set max_tokens=8000 for source chunks. That leaves room for the system prompt and any glossary we inject, while keeping the expected translation output under 4,096 tokens. The overlap of 200 tokens (about 150 English words) ensures the next chunk starts with a bit of context already seen, reducing boundary errors. A subtle improvement: we also split paragraphs into sentences using nltk.sent_tokenize before building chunks if a single paragraph is longer than the max token limit. That way we never have a chunk that exceeds the limit because of one giant paragraph. Carrying context across chunks Translation quality depends heavily on continuity. Names, places, and invented terms must stay consistent. Our approach is to include a running context in the prompt for each chunk. This context consists of: The last ~500 characters of the previous source chunk. The corresponding translation of that tail. A glossary of proper nouns extracted from the entire book. We build the glossary once before translation using spaCy: import spacy nlp = spacy.load('en_core_web_sm') def build_glossary(text: str, max_terms=100) -> str: doc = nlp(text[:100000]) # limit for speed names = set() for ent in doc.ents: if ent.label_ in ['PERSON', 'ORG', 'GPE', 'LOC']: names.add(ent.text) return ', '.join(sorted(names)[:max_terms]) Then, for each chunk translation, we prepend the glossary and previous context to the prompt: prompt = f'''Translate the following book text from English to {target_lang}. Maintain style, tone, and terminology. Use the glossary: {glossary_str} Previous translated context (for consistency): {prev_context} Source text: {chunk} ''' This two-part context (glossary + previous tail) dramatically reduced name inconsistencies in our tests. Initially we saw character names translated differently in later chapters; after adding the glossary, those errors dropped to near zero. Asynchronous orchestration with rate limiting Claude's API has rate limits (requests per minute and tokens per minute). To speed up translation while respecting those limits, we use anthropic.AsyncAnthropic with an asyncio.Semaphore. We typically run 4 concurrent requests for Claude Sonnet. Here's the core translation function: import asyncio import anthropic client = anthropic.AsyncAnthropic(api_key='...') async def translate_chunk(chunk: str, prev_context: str, glossary_str: str, target_lang: str, semaphore: asyncio.Semaphore) -> str: async with semaphore: prompt = f'''Translate the following book text from English to {target_lang}. Maintain style, tone, and terminology. Use the glossary: {glossary_str} Previous translated context (for consistency): {prev_context} Source text: {chunk} ''' for attempt in range(3): try: response = await client.messages.create( model='claude-3-sonnet-20240229', max_tokens=4096, temperature=0.3, system='You are a professional literary translator.', messages=[{'role': 'user', 'content': prompt}] ) return response.content[0].text except anthropic.RateLimitError: await asyncio.sleep(60 * (attempt + 1)) raise RuntimeError('Rate limit retries exhausted') The retry loop handles transient 429 errors. We also catch anthropic.APIStatusError for server-side overloads. The main loop processes chunks sequentially to maintain context, but the actual API calls are concurrent within each chunk? Actually, we process chunks one by one in order because each chunk's context depends on the previous translation. To get some parallelism, we can translate chunks in batches where each batch shares the same previous context, but we found that reduced quality slightly. For now, sequential chunk translation with a concurrency of 1 per book is simpler and only takes about 2–3 minutes for a 300-page book using Sonnet. If you need higher throughput, you could translate independent sections in parallel (e.g., different chapters) and then merge, but you lose the running context. Concrete performance and cost numbers For a 300-page novel (~120,000 tokens), our chunker produces around 16–20 chunks of 8,000 tokens with overlap. Translating to Spanish with Claude 3 Sonnet: Total input tokens sent: ~135,000 (including overlaps and prompt overhead). Total output tokens: ~110,000. Cost: $0.45 for input ($3/M) + $1.65 for output ($15/M) = ~$2.10 per book. Wall-clock time: 2–3 minutes with sequential chunks and no retries. We experimented with Claude 3 Haiku for a cheaper option ($0.25/M input, $1.25/M output). It cost about $0.35 per book but produced noticeably worse translations, especially for literary text with figurative language. For technical or non-fiction books, Haiku is acceptable; for novels, we stick with Sonnet. Handling PDF extraction problems PDF files are the wild west. Headers, footers, page numbers, and two-column layouts wreak havoc on chunking. We use PyMuPDF (fitz) because it gives us block-level text with bounding boxes, allowing us to filter out headers/footers based on vertical position. For EPUB, we use ebooklib to iterate over spine items and extract HTML, then strip tags with BeautifulSoup. This preserves chapter boundaries, which we use as natural chunk boundaries before applying token-based splitting. One hard-won lesson: always remove page numbers and repeated header text before tokenizing the full text. Otherwise, those artifacts become part of the chunks and get translated, producing garbage in the middle of chapters. Lessons learned and open questions What worked: Sentence- and paragraph-aware chunking with token overlap. Injecting a glossary and previous translation tail as context. Asynchronous retry for rate limits. Using PyMuPDF with positional filtering for PDFs. What didn't: Character-based splitting caused mid-sentence chunks. Parallel chunk translation without shared context led to name and style drift. Trusting output token limit blindly: some languages expand by 20–30%, so a 8,000-token source chunk can produce >4,096 tokens in German. We had to reduce source chunk size to 6,500 for such languages. Open question for the community: Have you found a reliable way to translate poetry, code, or tables inside long documents without breaking the surrounding narrative? We currently treat them as opaque blocks and translate them separately, but integration remains rough. Building LectuLibre taught us that long-form LLM translation is less about model capability and more about solid engineering around context management. If you're building something similar, start with robust chunking and context propagation—the model will handle the rest.