The first time an AI agent remembers something useful, it feels like product magic. The first time it remembers the wrong thing — a stale user preference, an old pricing rule, a deleted project, a support decision from six months ago — you realize the real problem is not making the agent smarter. The real problem is deciding what deserves persistence. A lot of agent architecture debates collapse into vague team arguments: “We need RAG.” “No, we need memory.” “Can’t the model just remember it?” “Should we store the conversation?” “Why did the agent use that old fact?” Those questions are usually missing the important one: Where does this piece of information belong, who owns it, how fresh is it, and when should it expire? For production AI agents, the answer is rarely “store everything.” The better answer is usually a division of labor: Tools provide live, authoritative state. RAG provides broad, versioned knowledge. Memory provides durable, scoped facts about users, tasks, or preferences. The context window holds temporary working state. Logs and traces hold auditability, not necessarily agent memory. The agent should store less than you think — but what it stores should be deliberate, scoped, reviewable, and expire when it stops being true. TL;DR Do not store live system truth in agent memory. Query it through tools. Use RAG for large, mostly read-only knowledge: docs, policies, runbooks, product knowledge. Use memory for durable, scoped facts: preferences, project context, user corrections, stable decisions. Do not store raw conversation transcripts as “memory” unless you have a retrieval, retention, and privacy plan. Do not store derived facts that can be recomputed cheaply. Keep session state ephemeral. Store tool schemas, permissions, and guardrails as governed artifacts. Build one context assembly layer that decides what information enters the model context and why. 📋 Table of Contents The Question Is Not “RAG or Memory?” 1. Store Live Truth in Tools, Not in the Agent 2. Use RAG for Large, Mostly Read-Only Knowledge 3. Store Durable User Facts as Scoped Memory 4. Keep Episodic Memory Small and Summarized 5. Don’t Store Derived Facts That Can Be Recomputed 6. Use the Context Window for Temporary Working State 7. Store Policies as Versioned Retrieval Content, Not Personal Memory 8. Tool Schemas and Permissions Are Also Stored Information 9. Build One Context Assembly Layer With Priorities Comparison Table: Where Should the Information Live? A Practical Storage Decision Framework The Question Is Not “RAG or Memory?” RAG, memory, and tools are not competing features. They answer different questions. RAG is best for retrieving knowledge from a corpus: documentation, policies, product guides, support articles, runbooks, internal wikis, codebase summaries, or legal guidelines. Memory is best for durable facts that should persist across sessions: user preferences, stable project context, explicit corrections, long-lived decisions, or summaries of past interactions. Tools are best for live state and authoritative actions: account balances, subscription status, order details, feature flags, permissions, calendar availability, ticket status, database records, or API-side truth. Context window is best for temporary state: the current task, the current form fields, the immediate user goal, the most recent tool results, and short-lived reasoning artifacts. The production mistake is not choosing the wrong one. It is blurring them together. For example: A user’s current plan should usually come from a billing tool, not from memory. A refund policy should usually come from versioned RAG content, not from a remembered chat snippet. A user’s preference for metric units may belong in memory. The fact that the user is currently editing a draft report belongs in session state. The user’s password never belongs in agent memory. Once you separate those categories, the architecture becomes much easier to reason about. 1. Store Live Truth in Tools, Not in the Agent Scenario: An agent remembers that a user is on the Pro plan. Two weeks later, the user downgrades. The agent still thinks the user has Pro features and gives incorrect answers or attempts actions the user is no longer allowed to perform. This is one of the most common failures in agent systems. Why it matters: Live system state changes constantly. Subscriptions change. Permissions change. Orders ship. Tickets close. Documents get deleted. Feature flags flip. Inventory runs out. If the agent stores that state as memory, it now has a second source of truth. Eventually, the two sources disagree. Solution: Use tools for authoritative, mutable state. If the information answers questions like: What is the user’s current plan? What is the current order status? Is this user allowed to perform this action? What is the current balance? Which projects are active? What is the latest version of this record? then the agent should usually call a tool, not rely on memory. from dataclasses import dataclass @dataclass(frozen=True) class Subscription: plan: str status: str renews_at: str features: set[str] class BillingClient: def get_subscription(self, user_id: str) -> Subscription: # In a real system, this calls your billing service. raise NotImplementedError def get_plan_context(user_id: str, billing: BillingClient) -> str: subscription = billing.get_subscription(user_id) return ( f"User plan: {subscription.plan}\n" f"Subscription status: {subscription.status}\n" f"Features enabled: {', '.join(sorted(subscription.features))}" ) The important part is not the code. It is the boundary: the agent does not “remember” the subscription. It asks the system of record. Why this works: Tools keep the agent aligned with the authoritative source. They also make permissions easier to enforce. The billing service can reject the request if the caller is not allowed to see the data. When memory may still be useful: You might remember that the user asked about upgrading last week. But the current plan itself should still come from the tool. ⚠️ Gotcha: If you cache tool results, treat the cache as a performance optimization with a short TTL, not as truth. 2. Use RAG for Large, Mostly Read-Only Knowledge Scenario: Your support agent needs to answer questions about refund policies, pricing rules, troubleshooting steps, and product limitations. The knowledge base changes occasionally, but not every second. You do not want to stuff all of that knowledge into the prompt. You also do not want the agent to “remember” policies from past conversations. Why it matters: Large knowledge corpora are too big to fit directly into context, and they change enough that hardcoded knowledge becomes stale. RAG gives you retrieval with source control. RAG is a good fit for: Product documentation. Internal policies. Support runbooks. API documentation. Legal or compliance guidelines. Codebase knowledge summaries. FAQ content. Historical tickets, if properly scoped and anonymized. Organization-specific procedures. The key is that RAG content should be treated as versioned knowledge, not casual memory. from dataclasses import dataclass from datetime import date @dataclass(frozen=True) class KnowledgeChunk: chunk_id: str source_id: str product: str locale: str status: str effective_date: date text: str def eligible_chunks( chunks: list[KnowledgeChunk], product: str, locale: str, as_of: date, ) -> list[KnowledgeChunk]: return [ chunk for chunk in chunks if chunk.product == product and chunk.locale == locale and chunk.status == "published" and chunk.effective_date <= as_of ] This is intentionally simple, but it shows the kind of metadata that matters: source_id: where the chunk came from. product: which product or plan it applies to. locale: language or region relevance. status: draft, published, deprecated. effective_date: when the knowledge became valid. Why this works: RAG allows the agent to ground answers in retrievable documents while still filtering by relevance, freshness, and authority. What RAG is not good for: It is not a good place for highly personal, mutable user state. A user’s current address, active project, or subscription status usually belongs in a tool or database, not in a vector store. 💡 Practical note: If your RAG system cannot tell you which document produced an answer, debugging becomes guesswork. Store source identifiers and retrieval metadata from the beginning. 3. Store Durable User Facts as Scoped Memory Scenario: A user tells the agent, “Always use UTC timestamps in reports.” The next week, the agent still uses local time because that preference was buried in a conversation transcript and never extracted as a durable fact. This is where memory is genuinely useful. Why it matters: Some information is not large enough to need RAG, not live enough to need a tool, and not temporary enough to belong only in the context window. Good examples of agent memory: The user prefers concise answers. The user’s organization uses metric units. The user’s default deployment environment is staging. The user corrected the agent about a naming convention. The user is working on a project named “billing-migration.” The user does not want automated emails. The user’s team uses main as the default branch. The problem is that “memory” often becomes a junk drawer. A production memory system needs structure. A useful memory record has: A scope. A key. A value. A source. A confidence level. A timestamp. An optional expiration. from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class MemoryRecord: memory_id: str scope: str key: str value: str confidence: float source: str created_at: datetime expires_at: datetime | None def select_memories( records: list[MemoryRecord], scope: str, now: datetime, limit: int = 8, ) -> list[MemoryRecord]: active = [ record for record in records if record.scope == scope and (record.expires_at is None or record.expires_at > now) ] active.sort(key=lambda record: (record.confidence, record.created_at), reverse=True) return active[:limit] The scope field is critical. Examples: user:12345:preferences workspace:analytics:conventions project:billing-migration:context conversation:support-ticket-987:summary A preference like “use UTC” may be user-scoped. A convention like “all migration scripts go in scripts/billing” may be project-scoped. A temporary detail like “the user is debugging a failing test” may be conversation-scoped and should expire quickly. Why this works: Structured memory lets you retrieve the right fact without dumping the whole conversation history into context. It also makes memory reviewable, editable, and deletable. What not to store here: Do not store secrets, tokens, passwords, payment card numbers, or sensitive personal data unless you have a very strong reason and proper controls. Memory is not a substitute for secure credential storage. 4. Keep Episodic Memory Small and Summarized Scenario: Your agent needs to know what happened in previous conversations. The simplest approach is to store every transcript and retrieve chunks of it later. Then the agent starts surfacing irrelevant details, repeating old issues, or bringing up resolved problems as if they are still open. Raw conversation logs are not the same as useful memory. Why it matters: Episodic memory — memory of events, conversations, and past interactions — can be valuable. But raw transcripts are noisy, long, repetitive, and often contain information you do not want the agent to keep using forever. Better episodic memory usually means storing summaries, outcomes, and follow-up state. from dataclasses import dataclass, field from datetime import datetime @dataclass(frozen=True) class EpisodeSummary: episode_id: str user_id: str summary: str outcome: str follow_up_needed: bool occurred_at: datetime source_message_ids: list[str] = field(default_factory=list) A good episode summary might look like this: User investigated a failing CI pipeline. The issue was caused by a missing environment variable. User fixed the CI configuration. No follow-up needed. That is often more useful than storing hundreds of raw messages. What episodic memory is good for: “What did we try last time?” “Was this issue resolved?” “Did the user ask for a follow-up?” “Which project were we working on?” “What decision was made?” “What was the outcome of the last incident?” What episodic memory is bad for: Long-term factual truth. Live account state. Sensitive personal data. Permanent storage of every user statement. Replacing audit logs or compliance records. Why this works: Summaries reduce noise and make retrieval more precise. They also reduce privacy risk by storing less raw content. 🔍 Why this matters: If your episodic memory contains everything, it will eventually surface something you wish it had forgotten. 5. Don’t Store Derived Facts That Can Be Recomputed Scenario: The agent remembers that a user has spent $1,250 this month. A week later, the user makes another purchase. The agent still uses the stale number because it was stored as a fact instead of being recomputed. Derived facts are dangerous because they look stable but are actually snapshots. Why it matters: Derived facts include things like: Total spend. Number of active projects. Average response time. Unresolved ticket count. Current usage quota. Feature adoption score. Remaining credits. Team member count. These can usually be computed from a system of record. If you store them, you now have to manage staleness, invalidation, and consistency. Solution: Recompute derived facts when needed, or cache them briefly if performance requires it. from dataclasses import dataclass from datetime import datetime, timedelta @dataclass(frozen=True) class CachedValue: value: object expires_at: datetime class SimpleCache: def __init__(self): self.store: dict[str, CachedValue] = {} def get(self, key: str) -> CachedValue | None: return self.store.get(key) def set(self, key: str, value: CachedValue) -> None: self.store[key] = value def get_monthly_spend( user_id: str, billing, cache: SimpleCache, now: datetime, ) -> float: cache_key = f"monthly_spend:{user_id}" cached = cache.get(cache_key) if cached and cached.expires_at > now: return cached.value spend = billing.get_monthly_spend(user_id) cache.set( cache_key, CachedValue( value=spend, expires_at=now + timedelta(minutes=5), ), ) return spend The cache is not memory. It is a short-lived performance layer. Why this works: It prevents the agent from treating a stale computation as a permanent truth. If the underlying system changes, the next recomputation picks up the change. When storing derived facts may be acceptable: If the derived fact is historical and immutable, such as “monthly spend for January 2026,” storing it may be fine. If it represents live state, prefer recomputation or a short TTL. 6. Use the Context Window for Temporary Working State Scenario: The user is filling out a multi-step request. The agent asks for a date, then a project name, then a timezone. If those temporary values are written into long-term memory, the agent may later reuse them in unrelated conversations. Not everything the agent sees deserves persistence. Why it matters: The context window is the agent’s short-term working area. It is appropriate for: The current user goal. The current draft. Missing required fields. Recent tool results. Temporary clarifications. In-progress selections. The current conversation language. The active document or record being edited. This information is useful, but only for the current task. from dataclasses import dataclass, field @dataclass class SessionState: current_intent: str | None = None missing_fields: dict[str, str] = field(default_factory=dict) draft_content: str | None = None confirmation_pending: bool = False active_project_id: str | None = None This kind of state can live in session storage, a conversation record, or server-side state associated with the interaction. The important part is that it should not automatically become durable agent memory. Why this works: Temporary state keeps the agent coherent within a task without polluting future tasks. Common mistake: Teams store every intermediate state in “memory” because it is easy. Later, the agent behaves strangely because it remembers half-finished tasks, abandoned preferences, or outdated form inputs. A good rule: If the information would be embarrassing or confusing in a different conversation, it probably should not be long-term memory. 7. Store Policies as Versioned Retrieval Content, Not Personal Memory Scenario: A support agent learns from one conversation that “we allow refunds after 90 days for enterprise customers.” That rule gets stored as a memory. Later, the official policy changes. The agent keeps using the old rule because it was remembered as a fact rather than retrieved from the current policy document. This is a subtle but important failure mode. Why it matters: Policies, rules, SLAs, pricing terms, compliance requirements, and internal procedures are organizational knowledge. They should be versioned, reviewed, and retrieved from an authoritative source. They should not become personal memories attached to a user or conversation. from dataclasses import dataclass from datetime import date @dataclass(frozen=True) class Policy: policy_id: str version: int status: str effective_date: date text: str def select_active_policy(policies: list[Policy], policy_id: str, as_of: date) -> Policy | None: candidates = [ policy for policy in policies if policy.policy_id == policy_id and policy.status == "active" and policy.effective_date <= as_of ] if not candidates: return None return max(candidates, key=lambda policy: policy.version) Why this works: The agent answers policy questions by retrieving the current approved policy, not by remembering a fragment from a previous chat. What belongs in policy retrieval: Refund rules. Security requirements. Data retention rules. Approval workflows. Escalation criteria. Support SLAs. Compliance constraints. Internal operational playbooks. What belongs in memory instead: A user-specific exception may be memory, but only if it is durable and properly scoped. For example: “This customer has an approved exception to the standard onboarding flow.” Even then, the exception should ideally be stored in a system of record and accessed through a tool, not only in agent memory. 🚨 Production warning: If a policy can change and the agent can act on it, policy retrieval needs versioning and auditability. 8. Tool Schemas and Permissions Are Also Stored Information Scenario: An agent tries to create a support ticket but sends the wrong fields. Or it attempts to delete a resource when it should only have read access. Or it calls a tool with an invalid enum value because the tool description was vague. Tool definitions are not just plumbing. They are information the agent uses to decide what it can do. Why it matters: Agents need to understand: What tools exist. What each tool does. What parameters are required. What values are valid. What side effects the tool has. What permissions are required. When the tool should not be used. This information should be stored as a governed registry, not improvised in prompts. from dataclasses import dataclass, field @dataclass(frozen=True) class ToolSpec: name: str description: str scopes: set[str] parameters_schema: dict irreversible: bool = False requires_confirmation: bool = False CREATE_TICKET_TOOL = ToolSpec( name="create_support_ticket", description=( "Create a support ticket for the current user. " "Use this only after confirming the issue cannot be resolved directly." ), scopes={"support:write"}, parameters_schema={ "type": "object", "properties": { "title": {"type": "string"}, "description": {"type": "string"}, "severity": { "type": "string", "enum": ["low", "medium", "high"], }, }, "required": ["title", "description", "severity"], "additionalProperties": False, }, irreversible=False, requires_confirmation=False, ) DELETE_PROJECT_TOOL = ToolSpec( name="delete_project", description=( "Delete a project permanently. This action cannot be undone. " "Use only after explicit user confirmation." ), scopes={"project:delete"}, parameters_schema={ "type": "object", "properties": { "project_id": {"type": "string"}, }, "required": ["project_id"], "additionalProperties": False, }, irreversible=True, requires_confirmation=True, ) This gives the agent and the surrounding application a clear contract. Why this works: The agent is less likely to invent parameters, misuse destructive tools, or call actions it does not have permission to perform. The application can also enforce scopes and confirmation rules before execution. What to store with tool information: Tool name. Human-readable description. Parameter schema. Required scopes. Side-effect class: read, write, delete, external send, payment, etc. Confirmation requirements. Rate limits or quota information. Whether the tool is available in sandbox, production, or both. This is one of the most overlooked parts of agent memory architecture. Tool knowledge is not just configuration. It is part of the agent’s operational context. 9. Build One Context Assembly Layer With Priorities Scenario: Your agent has RAG results, user memories, tool outputs, policy documents, session state, and previous episode summaries. Everything seems relevant. The context window fills up. The model receives conflicting information. Some of it is stale. Some of it is too broad. Some of it should not be in this conversation at all. This is where many agent systems break. Why it matters: The question is not only what the agent stores. It is what the agent chooses to place into context for a specific request. A context assembly layer decides: Which tool results are required. Which policies are active. Which memories are in scope. Which RAG chunks are relevant enough. Which session state is still valid. What gets truncated or omitted. What requires human approval before inclusion. from dataclasses import dataclass @dataclass(frozen=True) class ContextItem: kind: str priority: int text: str required: bool = False def estimate_tokens(text: str) -> int: # Rough estimate. Use a tokenizer for precise budgeting. return max(1, len(text) // 4) def assemble_context( items: list[ContextItem], token_budget: int, reserve_tokens: int = 500, ) -> list[ContextItem]: selected: list[ContextItem] = [] used_tokens = 0 available = max(0, token_budget - reserve_tokens) ordered = sorted(items, key=lambda item: (not item.required, item.priority)) for item in ordered: tokens = estimate_tokens(item.text) if item.required or used_tokens + tokens <= available: selected.append(item) used_tokens += tokens return selected The exact token estimate is less important than the discipline of budgeting and prioritization. A reasonable priority order might be: Active task contract and output constraints. Live tool results required for the current action. Active policy or safety guidance. Scoped user memory directly relevant to the request. RAG chunks from authoritative sources. Episodic summaries from previous interactions. Background knowledge or optional examples. Why this works: It prevents the context from becoming an uncontrolled dump of everything the agent has ever seen. It also makes failures easier to debug because you can inspect what was included and why. What to log in production: You do not necessarily need to log every raw prompt forever, but you should be able to reconstruct: Which tool results were used. Which memory records were included. Which R chunks were retrieved. Which policy version was used. What was truncated. Which user or workspace scope was active. That traceability is what turns “the agent did something weird” into a debuggable system issue. Comparison Table: Where Should the Information Live? Information Best Place Should the Agent Store It? Example Current subscription plan Tool No, query it Billing API Refund policy RAG Store as versioned docs Policy database/vector index User prefers dark mode Memory Yes, if durable User preference store Current form draft Context/session state Temporarily only Multi-step wizard Last support conversation outcome Episodic memory Yes, summarized Episode summary Current order status Tool No, query it Orders API API credentials Secret manager Never in agent memory Vault/KMS Internal security policy RAG/policy store Versioned retrieval Compliance docs User’s active project Tool or scoped memory Depends on source of truth Project service Temporary clarification Context window No persistent memory “Use March data” Tool schemas Tool registry Yes, governed Action definitions Derived monthly total Recompute/cache Short TTL only Spend calculation A useful mental model: If it changes often and must be correct, use a tool. If it is broad knowledge, use RAG. If it is durable and user-specific, use scoped memory. If it is only relevant now, use session context. If it is sensitive, do not store it unless absolutely necessary. A Practical Storage Decision Framework When deciding whether an AI agent should store a piece of information, walk through these questions in order. 1. Is this the live source of truth? If yes, do not store it in memory. Access it through a tool or API. Examples: Account status. Permissions. Inventory. Order state. Feature flags. Current configuration. 2. Is this broad organizational knowledge? If yes, use RAG with metadata and versioning. Examples: Documentation. Policies. Runbooks. Product guides. Internal procedures. 3. Is this a durable fact about the user, workspace, or project? If yes, store it as structured memory with scope and expiration. Examples: Preferences. Corrections. Stable project conventions. Long-lived decisions. Explicit user instructions. 4. Is this just temporary task state? If yes, keep it in session state. Examples: Current draft. Missing fields. Active selection. Recent clarification. In-progress confirmation. 5. Is this derived from other data? If yes, recompute it or cache it briefly. Examples: Totals. Counts. Averages. Scores. Quota remaining. 6. Is this sensitive? If yes, minimize storage. Do not store: Passwords. Tokens. Secrets. Payment card numbers. Sensitive personal data without a clear product and legal basis. Information the user would reasonably expect to remain temporary. 7. Can this be reviewed, corrected, or deleted? If the answer is no, be careful. Production memory should be inspectable. Users and operators should be able to answer: Why does the agent know this? Where did it come from? When was it created? Can it be edited? Can it be deleted? Does it expire? Which scope does it apply to? If you cannot answer those questions, the information probably should not be in long-term memory yet. The best agent architecture is not the one that remembers the most. It is the one that knows where each fact belongs, how fresh that fact is, and when it should stop using it.