Six ways of remembering what an agent has done, measured on one 14B model over 600-step jobs: the whole chat, a generic summary, a summary that writes the state out, a sliding window, retrieval, and a host-held state shown fresh each step. With the host's check in the loop, all six ended with the exact state on every step they ran. They differed in cost: 0.30 s and 21.8 joules a step for the host-held state, 0.55 to 0.66 s and 39.4 to 48.0 joules for summaries, a window and retrieval, 1.72 s and 123.1 joules for the whole chat. They also differed on something the host-held state did not win: how often the model's own reply was right.
Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.
What is the fair objection to "keep the whole chat"?
Nobody keeps the whole chat for long. Tools summarise it, cut it to a window, or fetch the old turns that matter. So Test 44 puts each of those against a host that holds the state and shows the model the exact state every step.
What was compared?
Qwen2.5-14B-Instruct (4-bit) on vLLM 0.31.0, one NVIDIA L4, prefix caching on, greedy decoding, no retry, three jobs of 600 steps (1,800 steps per arm). In every chat arm the first message shows the whole state and the first request; later messages are only the requests, and the model's replies stay in the chat.
| Arm | What it is |
|---|---|
| Exact state (host-held) | each step: fresh prompt with the state the host holds and the request; the host checks and repairs the reply |
| Whole chat | the chat kept whole until its next prompt would pass 16,000 tokens |
| Summary | at 4,000 tokens the model summarises the chat and the summary replaces it |
| Summary that writes the state out | the same, but the model is asked to write out the complete current list of services and edges |
| Sliding window | first turn pinned, then the last 12 turns |
| Retrieval | first turn pinned, the last 4 turns, and the 8 earlier turns sharing the most coordinates with the request |
What came out?
| Arm | Steps | Model's own reply right | Held state exact (with the host's check) | Applied as written, exact | Computed prompt tokens a step | Seconds a step | Joules a step |
|---|---|---|---|---|---|---|---|
| Exact state | 1,800 | 81.89 % | 100.0 % | 0.39 % | 149.2 | 0.3034 | 21.8 |
| Whole chat | 1,160 | 89.83 % | 100.0 % | 0.69 % | 1,755.1 | 1.7224 | 123.1 |
| Summary | 1,800 | 84.00 % | 100.0 % | 0.44 % | 42.8 | 0.5477 | 39.9 |
| Summary that writes the state out | 1,800 | 85.28 % | 100.0 % | 0.44 % | 45.3 | 0.6255 | 45.5 |
| Sliding window | 1,800 | 85.33 % | 100.0 % | 0.44 % | 489.3 | 0.5503 | 39.4 |
| Retrieval | 1,800 | 93.83 % | 100.0 % | 0.44 % | 496.2 | 0.6631 | 48.0 |
The whole chat stopped on its 16,000-token rule at step 418, 367 and 375 of 600 in the three jobs. Computed tokens include the summary calls; joules are the board's energy divided by the steps run (mean power 70.4 to 72.0 W in every arm).
What does the table say?
- The host's check, not the memory method, held the state. The same replies applied as written left the state right on 0.39 to 0.69 percent of steps, and not even a well-formed graph on 98.3 to 99.6 percent of them. With the host's gate and repair, all arms were exact on every step.
- The host-held state was the cheapest way to run, in time and energy. It was the fastest arm (0.3034 s) and the lowest in joules (21.8). Summaries computed fewer new tokens (42.8 and 45.3 against 149.2), but the summary calls took the time: 494.9 of the 985.8 seconds of the summary arm were 21 summary calls.
- Showing the exact state did not make the model's own reply right more often. The model's own step was right on 81.89 percent with the exact state against 84.00 to 93.83 percent for the chat arms. On deletes of a service with edges, where the reply must list every edge, it was right on 21.63 percent against 30.77 to 73.32. The host's repair completes the missing edges, which is why the held state stayed exact.
- Asking a model to write the state down did not work here. None of the 22 summaries that were asked to write the state out equalled the true state (19 differed, 3 stopped at the 1,000-token limit before the edge list).
What is not shown?
One card, one 14B quantised model, greedy decoding, synthetic service graphs, three jobs of one family and 600 steps. The summary prompts, the window size (12 turns) and the retrieval rule were the author's single choices and were fixed before the run; a better summary prompt, a longer window or embedding-based retrieval would move those arms. The whole chat ran under engine memory warnings late in the arm, so its seconds are read with that in mind. The whole chat here shows the state once; the heavier chat that shows it every message is in the context-window page of this series.
Check it
The test page, with its predictions written before the run and every reading below a line, is docs/MEMORY_ARMS_TEST.md in the repository, with the saved summaries in docs/results/axi_memory_arms_summaries.json: github.com/gaurshalendra2412/axi-runtime. White paper: doi.org/10.5281/zenodo.23958737. Ten open-model tests: doi.org/10.5281/zenodo.24319561.
Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.