Ankita AI is a desktop AI assistant by Krish Verma, and it has a memory problem every long chat hits: the context window is finite, and at some point the oldest turns have to go. The usual answer is trimming — delete the oldest messages and move on. I built something else instead: when a chat reaches 36 messages, Ankita summarizes the turns trimming would delete into a single 2,000-byte recap and keeps the 10 most recent messages untouched. The history shrinks; the facts survive.

The pain: trimming deletes decisions

Ankita trims with trimMessages(), which is careful about one thing: it never splits an assistant tool call from its tool replies. But careful deletion is still deletion. In a long session — say, an afternoon of debugging where you decided on a fix, named three files, and left a task half done — the oldest turns are exactly the ones holding the decisions. Trim them and the companion forgets why it did what it did, then re-asks, re-does, or contradicts itself. The website is ankita-sable.vercel.app and the code is on GitHub if you want to read along.

The fix lives in src/core/agent.mjs, in compactHistoryIfNeeded(), which runs inside trimHistory() before any trimming happens. It is a recap, not a deletion: the old turns are summarized by the model into one note, and that note takes their place in the history.

The numbers that decide when it fires

The trigger is deliberately boring arithmetic, all constants at the top of agent.mjs:

  • 16 messages minimum (COMPACTION_MIN_MESSAGES). Below that, a chat is too short to be worth an extra model call.
  • Trigger at 36 messages: historyMessages defaults to 40, minus a margin of 4 (COMPACTION_TRIGGER_MARGIN). The recap runs shortly before trimming starts, so facts survive the first drop instead of being rescued after it.
  • Keep the 10 most recent messages (COMPACTION_KEEP_RECENT), and grow that count while the oldest kept message is a tool reply — the live task keeps full fidelity, and no tool exchange is ever split.
  • At least 6 old turns (COMPACTION_MIN_OLD) must be worth summarizing, and 8 new messages (COMPACTION_COOLDOWN_TURNS) must arrive before another recap is earned. No recap spam.

The summarizer input is capped hard: 800 characters per message (COMPACTION_MESSAGE_CHARS), 12,000 bytes total (COMPACTION_TRANSCRIPT_BYTES). Tool calls are flattened to [called: write_file, run_command] so the summary knows what actually ran. The prompt asks for at most 300 words and names exactly what to preserve: user goals, decisions made, files and paths touched, tools used and their key results, unresolved tasks, and constraints. It ends with two lines I care about most: "Do not invent facts. Return only the summary text, no preamble."

The reply is capped at 400 tokens (COMPACTION_MAX_TOKENS) and then cut to 2,000 bytes (COMPACTION_REPLY_BYTES) — in bytes, not characters, so multi-byte scripts like Hinglish or Hindi cannot push the recap past the history budget it exists to protect.

Two small decisions that carry the design

First, the recap is inserted as a user-role message, prefixed with Compacted summary of earlier conversation:. That is not cosmetic: some providers require the first non-system turn to be the user, and a summary wearing the assistant's voice would break that shape. It also means the desktop transcript can show the recap as a labelled note, not as something you typed.

Second, failure is soft. If the summarizer request fails or returns nothing, compactHistoryIfNeeded() prints a one-line note and returns — plain trimming proceeds as if compaction never existed. A recap that cannot be written must never be the thing that breaks the chat. A localhost round trip confirmed the wiring: zero provider requests while compaction is off, exactly one recap request while it is on.

The setting is on by default in the desktop app (Settings → Model → Long conversations), while the CLI stays opt-in with CONTEXT_COMPACTION=on — and an explicit value in .env always wins over the app default. That split is deliberate: the desktop is where the long chats happen.

What I am not claiming

This is covered by test/core/context-compaction.test.mjs (8 tests), but no live third-party provider recap has been run yet, so recap wording, cost, and latency against a real model are unmeasured. And the honest hole is structural: the recap is written by the same model that will later read it as fact. "Do not invent facts" is an instruction, not a verification pass — if the summary hallucinates a decision, nothing in the pipeline catches it, and the recap then sits in history wearing the user's role, indistinguishable from something I actually said.

Tell me I'm wrong: is a 300-word summary of 12 KB of transcript trustworthy enough to act on, or should the recap be verified against the transcript before it is allowed to stand as history — and if you would verify it, what would you check it against, the raw turns or the decisions it claims?

— Krish Verma, creator of Ankita AI