Why Memory Matters for Agents

LLMs are stateless by default. Every time you call the API, the model starts fresh. For a simple Q&A bot, that's fine. For an agent that needs to:

• Remember what a user said 3 sessions ago
• Build a cumulative understanding of a project over weeks
• Learn from past mistakes and not repeat them
• Maintain consistent persona and context across interactions

...you need memory infrastructure. Without it, your agent is perpetually amnesiac.

The 4 Types of Agent Memory

1. In-Context Memory (Short-term)

The simplest form: the conversation history in the current context window. Every message exchanged in the session lives here.

• Capacity: Limited by context window (8K–1M tokens depending on model)
• Persistence: Disappears when the session ends
• Best for: Single-session tasks, debugging, simple chatbots

Management tip: For long conversations, use sliding window (drop oldest messages) or summarization (compress old turns into a summary) to stay within limits.

2. External Memory (Long-term, Semantic)

Store memories as vector embeddings in a vector database. At query time, retrieve the most relevant past memories and inject them into context.

• Capacity: Unlimited — millions of memories
• Persistence: Permanent until deleted
• Best for: Cross-session continuity, personalization, knowledge accumulation

Best tools:

• Mem0 — Purpose-built agent memory layer. Automatic memory extraction + retrieval. Drop-in replacement for message history.
• Letta (MemGPT) — Memory management as a first-class agent feature. Handles memory paging automatically.
• Zep — Long-term memory for LLM apps. Handles session management, fact extraction, and semantic search.

3. Episodic Memory (Event-based)

A structured log of past events, conversations, and actions. Think of it as the agent's diary — a timestamped record of what happened, when, and what the outcome was.

• Capacity: Bounded by storage (typically database)
• Persistence: Permanent structured records
• Best for: Audit trails, learning from past actions, "what did we do last time?"

Implementation: Store each agent run as a structured record (input, actions taken, tool calls made, output, success/failure, duration). Use LangSmith or Langfuse to capture this automatically.

4. Procedural Memory (Skill-based)

Encoded knowledge about how to do things — typically represented as system prompts, tool definitions, few-shot examples, or fine-tuned model weights.

• Capacity: Limited by context / model size
• Persistence: Updated through prompt editing or fine-tuning
• Best for: Consistent behavior, skill improvement, domain specialization

In practice: Your system prompt is your agent's procedural memory. Treat it like code — version it, test it, improve it based on observed failures.

Comparison Table

Type	Where stored	Lasts	Use case
In-context	LLM context	Session only	Short conversations
External (vector)	Vector DB	Permanent	Cross-session personalization
Episodic	SQL / NoSQL DB	Permanent	History, audit, learning
Procedural	Prompt / weights	Until updated	Skills, behavior patterns

Recommended Memory Stack

For a production personal assistant agent:

1. In-context: Last 20 messages, always included.
2. External (Mem0): Top 5 relevant memories retrieved per turn.
3. Episodic (Langfuse): Every session logged, searchable.
4. Procedural (system prompt): Updated weekly based on failure patterns.

Memory is the difference between an agent that feels like a tool and one that feels like a colleague. Invest in memory infrastructure early — retrofitting it is painful.

The Memory Hierarchy in Practice

When an agent starts a turn, here's the priority order for what goes into context:

1. System prompt (procedural memory) — always included
2. Retrieved relevant memories (external memory) — top K
3. Recent conversation history (in-context memory) — last N turns
4. Current user message

Everything above gets compressed/truncated if you exceed the context window. The system prompt and current message are always preserved.

Explore all memory tools at AgDex — indexed with descriptions, pricing, and links.