Day 5 of 7 — The Agentic Engineer’s Field Notes

I want to start today with a confession.

Of the four subsystems we’ve been walking through — model, tools, memory, loop — the one I am least confident about is memory. Not because we don’t have techniques. We have plenty. The problem is that none of them are fully right, the trade-offs are subtle, and most of the polished “memory features” you’ve seen demoed are doing the easy 20% and avoiding the hard 80%.

If you are building agents and you feel like memory is harder than it should be: you are not wrong. Today I want to give you the honest version of where memory actually stands in 2026, what works in production, and what to be skeptical of when someone tells you they’ve solved it.

What memory has to solve

Start with the problem statement, because it is more subtle than “remember stuff.”

Modern LLMs are stateless. Every call begins from zero. The model has no memory of yesterday’s conversation, your user’s preferences, or the half-finished plan from twenty minutes ago. Whatever you want it to know, you have to put into the prompt — every single call.

This is fine for a chatbot. For an agent, it is a structural problem. Agents do work that spans turns, sessions, tasks, and sometimes weeks. The agent that helped you draft a contract on Monday should — when you come back on Friday — know that you exist, that the contract was for a specific client, that you preferred the 30-day termination clause, and that the legal team flagged section 4 for revision.

None of that is in the model’s weights. All of it has to be reconstructed, every call, from somewhere. That somewhere is what we mean by memory in agents.

The OS analogy is not a stretch

Yesterday I mentioned Karpathy’s framing of LLMs as a new kind of operating system, where the context window is the RAM. Memory in agents is precisely the disk-and-swap layer of that operating system, and the analogy holds far further than it has any right to.

The Letta team (the same team that originally published MemGPT) has run with this analogy explicitly. Their architecture treats agent memory as three tiers, modeled on a real operating system’s memory hierarchy:

  1. Core memory — small, persistent blocks that live in the context window. Persona, user preferences, the current task’s pinned facts. Functions as RAM. Fast, expensive, finite.
  2. Recall memory — the complete history of past interactions, persisted automatically to disk, searchable on demand. Functions as a disk cache. Cheap, slower, broader.
  3. Archival memory — explicitly formulated knowledge stored in external databases (vector or graph). Processed and indexed information, distinct from raw history. Functions as cold storage. Cheapest per byte, slowest to access, effectively unbounded.

The crucial property of this architecture is that the agent can move data between tiers itself, using tools — memory_replace, memory_insert, archival_memory_search, conversation_search. The model is not a passive consumer of memory. It is a memory manager.

This shift — from “memory is a thing that gets injected into the prompt” to “memory is a hierarchy the agent operates over with tools” — is, in my view, the most important architectural shift in stateful agent design over the past two years.

The three kinds of memory, made concrete

Even if you are not using Letta, the three-tier model is worth carrying with you because it forces you to keep different kinds of memory distinct. Conflating them is the root of most memory bugs I’ve seen in production.

Working memory is the stuff the agent is using right now — the current conversation, the result of the tool call it just made, the plan it is partway through executing. This lives in the context window and is what we covered on Day 3 under context engineering.

Episodic memory is the record of what has happened. Last Tuesday at 3 PM the user asked us to summarize a meeting transcript. We produced this output. They corrected this part. Episodic memory is dated, ordered, conversational — essentially a journal of past runs.

Semantic memory is durable knowledge about the world. The user prefers SQL examples in PostgreSQL syntax. The team uses two-week sprint cycles. The customer’s primary contact is Sarah, in legal. Semantic memory is not tied to a specific moment — it’s a fact that should be available whenever it’s relevant.

These map roughly onto Letta’s tiers, but the mapping is not exact, and that’s the point. Different kinds of facts have different storage shapes, different retrieval patterns, and different decay properties. Treat them as one bag and you will keep finding bugs you can’t classify.

Why most “memory features” are theater

The dominant pattern, especially in product memory features that get demoed at conferences, is this: take everything the user says or the agent does, embed it into vectors, store it in a vector database, and on every new query do a top-k semantic search to retrieve “relevant memories” to stuff into the prompt.

This is fine for a demo. It collapses in production for predictable reasons.

It cannot handle contradictions. The user said in March that they prefer Python. In June they said they’re switching the team to Go. A vector store doesn’t know the second statement supersedes the first — it’ll happily retrieve both, and the agent has to figure it out.

It cannot handle retractions. “Forget what I said about the deadline” stores another vector, but the original “deadline is March 1” is still in the index, will still be retrieved by similarity, and will still confuse the agent.

It cannot handle salience. “I have two cats” and “I work in healthcare compliance” should not have the same weight in every interaction. The first is trivia; the second is mission-relevant. Pure semantic similarity treats them the same.

It cannot handle decay. “The user is using our v2 API” was true a year ago. It is no longer true. There is no natural mechanism in a vector store to expire stale facts.

These are not edge cases. They are the normal operating regime for any agent that runs longer than a few sessions with a real user. If your memory system can’t handle them, your agent will get worse over time, not better, as the noise accumulates.

What does work

The honest answer is that no single technique solves memory. The teams that have shipped memory-rich agents in production are using layered systems where each layer plays one role well.

Here is the layering I have seen hold up.

A scratchpad for the current task.

Before you reach for any external memory store, give the agent a place to write notes to itself during the run. This can be as simple as a file in a temp directory, or a structured “scratch” section in the agent’s working memory. The agent uses it to track sub-goals, intermediate decisions, things it wants to remember across loop iterations.

A lot of what looks like a memory problem is actually a within-task working-memory problem. The agent forgets where it was in the plan because the plan was implicit. Writing it down — explicitly, in a scratchpad the agent reads back from each turn — solves a surprising fraction of failures that get blamed on long-term memory.

File-based memory for durable preferences.

Underrated, in my view: just let the agent maintain plain-text files of facts it wants to keep.

This sounds primitive next to vector stores and graph databases. It has properties the fancy approaches lack. Files are editable — the agent can update a fact, retract a fact, restructure its own notes. Files are human-readable — you can audit what the agent thinks it knows about a user, edit it manually, ship it to staging. Files are grep-able — retrieval can be exact-match for things where exact match is what you want, semantic only when needed.

Claude Code’s CLAUDE.md pattern is one example: a project-specific markdown file that captures durable context the agent should have on every run. Letta's memory blocks are a more elaborate version of the same insight: structured, editable memory regions that the agent reads from, writes to, and reasons over explicitly.

The lesson generalizes. If a human reading the agent’s notes wouldn’t be able to make sense of them, the agent probably can’t either. Human-readable, structured, editable memory is a feature, not a primitive limitation.

Episodic logs you can replay.

For “what happened” memory, store full conversation logs — actual transcripts, not summaries — and pull them on demand. When the agent needs to know about Tuesday’s session, retrieve the transcript of Tuesday’s session, possibly summarized at retrieval time, and let the model read it.

This is more expensive than a vector hit, but it has the property of fidelity. The model gets to see what actually happened, in order, with all the detail it needs. Combined with a lightweight index — by date, by user, by topic — you can hit episodic memory only when it’s actually needed, not on every turn.

Recursive summarization for long horizons.

Long-running agents need to forget the right way. The technique that has converged across multiple production teams is recursive summarization: as messages roll out of the active window, they get summarized; old summaries themselves get summarized periodically; the result is a hierarchy of compressions where recent context is high-fidelity and ancient context is a brief synopsis.

This is what Letta does, and what Claude Code’s auto-compact does at the task level. The output is an agent whose memory of “what I did half an hour ago” is detailed, “what I did this morning” is summarized, and “what I did last week” is one paragraph — which is roughly how a human’s working memory of a project actually behaves.

Vector stores for true semantic recall.

Vectors are not useless. They are great for the case they were designed for: “find the document that is similar in meaning to this query.” Treat them as one tool in the memory toolkit, not the entire toolkit.

Use them for: searching across a large corpus of past content where you don’t know the exact query in advance. Looking up “have we ever discussed something like this before?” Finding analogous past tasks.

Don’t use them for: tracking the user’s current preferences. Storing the agent’s current plan. Recording structured facts. Anything that needs precision, mutation, or expiry.

The agent as memory manager.

The most powerful pattern, and the one I think will dominate over the next few years, is making the agent itself responsible for managing its memory.

This means: the agent has tools to read memory, write memory, update memory, and delete memory. After each task it considers what’s worth keeping. When it starts a new task it queries what’s relevant. The agent, not the framework, decides what to remember and what to forget.

This is the direction Letta has formalized — the agent has access to memory_replace and memory_insert for in-context blocks, plus archival_memory_insert and archival_memory_search for external storage. They have also pushed an interesting idea further: sleep-time agents, specialized agents whose job is to do memory consolidation between sessions. While the user-facing agent is idle, a sleep-time agent reads the recent traces, decides what's worth promoting to long-term memory, what should be deduplicated, what should be summarized further. It is the agentic equivalent of sleep doing memory consolidation in biological brains.

The trade-off: you trust the agent more. If the agent decides a fact isn’t worth keeping and was wrong, you’ve lost it. You’ll need observability into memory operations the same way you instrumented tool calls in Day 4.

The litmus test for a memory system

When evaluating any memory architecture — yours, a vendor’s, a framework’s — run it through these four scenarios:

  1. Contradiction. The user changes their mind. Does the system correctly prefer the new information without losing the audit trail of the old?
  2. Retraction. The user explicitly says “forget that.” Does the system actually forget, or does it just store another fact saying “forget that”?
  3. Decay. A fact that was true a year ago is no longer relevant. Does the system have a mechanism — agent-driven, time-based, or otherwise — to deprioritize or expire it?
  4. Salience. Two facts retrieved by similarity have different importance. Does the system have a way to weight them, or does it treat all retrievals equal?

A memory system that fails any of these will produce an agent that subtly degrades over time. The degradation is hard to spot from a single test — which is why so many memory systems look fine in demos and rot in production.

The teams that get this right invest in memory the same way they invest in their data layer. Schemas. Migrations. Audit logs. The discipline is database engineering with an LLM in the loop, not “magic AI memory.”

A practical place to start

If you are building your first agent and feel paralyzed by all of this, here is the minimal memory stack I’d recommend:

  1. A scratchpad file the agent reads and writes within the current task.
  2. A user-profile file, plain markdown, containing durable preferences and facts. The agent has tools to read it, edit it, append to it. Ship it with sensible defaults.
  3. An episodic log of past sessions, stored as transcripts, indexed by date and topic. Retrieval is on-demand via a tool, not always-on.
  4. Recursive summarization at task boundaries to keep the working context dense.
  5. A vector store only if you have a corpus that warrants one, and even then only as one of several retrieval tools.

Build that. Get an agent shipping. Watch where it fails. Add complexity only where the failure pattern justifies it. Start with files. Reach for vectors when you must. Reach for graphs when you really must.

When you outgrow this stack, Letta is the most mature open option for the next step up. Their hierarchical memory architecture and self-editing tools are state of the art for stateful agents, and worth studying even if you don’t adopt them directly.

Tomorrow: the discipline that decides whether your agent improves or rots

Once you have a system — model, tools, memory, loop — the question that determines your future is: how do you know it’s getting better?

I have watched teams ship agents and then run them blind for months. They fix one bug. The fix introduces three more. Nobody notices because nobody is measuring. The model gets upgraded. Half the agent’s behaviors change overnight. The team finds out from a user complaint.

The way out is evals. Real evals, not “we ran some test prompts and they looked OK.” A discipline of measurement that, in Hamel Husain’s framing, accounts for 60–80% of development time on production AI systems. Hamel and Shreya Shankar have basically codified the practice over the past two years, and tomorrow I’m going to walk through it: error analysis, the open-coding/axial-coding methodology, why binary judgments beat Likert scales, and the single habit that separates the teams whose agents quietly improve every quarter from the teams whose agents silently rot.

If you skip any day in this series, do not skip tomorrow. Memory is hard. Evals are boring. Boring beats hard.

See you then.

Day 5 of 7. We’ve covered: [_the mental shift](https://medium.com/@codewithrashid/why-most-agents-never-survive-production-and-the-mental-shift-that-fixes-it-ddffd932e1fb), [_agent anatomy](https://medium.com/@codewithrashid/the-anatomy-of-an-agent-that-actually-works-c4dd996ea4b9)_, [_context engineering](https://medium.com/@codewithrashid/context-engineering-has-eaten-prompt-engineerin-eb3a869c0362)_, [_tool design](https://medium.com/@codewithrashid/your-agent-is-only-as-good-as-its-tools-c0c431dbf795)_, and now memory. Tomorrow: evals — the boring discipline that separates winners from rotters._

Sources & further reading:

First published at codewithrashid.com.