If your AI agent loses all state the moment a conversation ends, you haven't built an assistant—you've built a stateless chatbot.

When developers first start building LLM applications, the standard approach to memory is appending every chat message into a growing array and passing it into the model on every turn. While this works for basic multi-turn conversations, it quickly breaks down in production.

Context windows fill up fast. Latency increases with token count, API charges scale exponentially, and models suffer from context degradation, frequently ignoring system rules when buried under thousands of tokens of raw transcript. Trying to solve multi-session memory by stuffing more tokens into the prompt is an anti-pattern.

To build production-grade agents that remember user preferences over days or months, you need a layered memory architecture: short-term context window management paired with long-term persistent storage.

Understanding the Architectural Split

An effective agent memory system divides responsibility into two primary layers:

  1. Short-Term Memory (Context Window): Resides in the active execution context of your application server. It tracks current thread state, immediate multi-turn back-and-forth, and active task progress. It is low latency and cheap to access, but ephemeral. It dies when the thread closes.
  2. Long-Term Memory (External Storage): Resides outside the model prompt in a SQL database, key-value store, or vector database. It contains extracted user facts, cross-session preferences, and historical summaries. It requires an network hop to read, but survives across sessions.

Step 1: Manage Short-Term Memory with Sliding Windows

Instead of sending the complete message payload on every turn, truncate the message array dynamically while preserving core system instructions.

Here is a backend pattern to bound short-term context window size:

def build_short_term_context( system_prompt: str, message_history: list[dict], max_recent_turns: int = 5 ) -> list[dict]: """ Maintains a fixed system prompt while keeping only the N most recent user/assistant message turns. """ payload = [{"role": "system", "content": system_prompt}] # Calculate the number of raw messages to retain (2 messages per turn) cutoff = max_recent_turns * 2 recent_history = message_history[-cutoff:] if cutoff > 0 else [] payload.extend(recent_history) return payload 

Step 2: Persist Facts to Long-Term Storage

As interactions occur, do not rely on raw transcripts for long-term recall. Parse incoming messages for key facts—such as technical preferences, location details, or recurring user tasks—and store them explicitly.

Vector databases are ideal for unstructured memory retrieval. Below is a implementation showing how to write and retrieve context using Python and ChromaDB:

import chromadb # Initialize persistent memory storage client = chromadb.PersistentClient(path="./agent_memory_db") memory_collection = client.get_or_create_collection(name="user_knowledge") def store_user_fact(user_id: str, fact: str, fact_id: str): """Saves an extracted user preference into the vector store.""" memory_collection.upsert( documents=[fact], metadatas=[{"user_id": user_id}], ids=[f"{user_id}_{fact_id}"] ) def fetch_relevant_memories(user_id: str, current_query: str, limit: int = 3) -> str: """Queries long-term memory for facts matching the current query.""" results = memory_collection.query( query_texts=[current_query], n_results=limit, where={"user_id": user_id} ) documents = results.get("documents", [[]])[0] if not documents: return "" formatted_facts = "\n".join([f"- {doc}" for doc in documents]) return f"Retrieved User Context:\n{formatted_facts}" 

Step 3: Combine Layers into a Single Payload

Before making an inference call, query your persistent vector storage using the user's latest input. Inject only the top relevant facts into the system prompt alongside the truncated short-term message buffer.

This gives your model access to critical preferences established weeks ago without cluttering the prompt window with unnecessary context.

Summary

Relying on context window length as a replacement for real database architecture creates slow, expensive agents. By separating your runtime context from persistent storage, your agents gain long-term intelligence while keeping response latency and token costs minimal.

About me

Tanmay builds AI agents and automation for businesses and is open to freelance projects. Feel free to connect on LinkedIn.