Most AI agent memory implementations look great in demos and fall apart in production. Here’s what I’ve learned building memory systems that actually work.
The Problem
Every agent framework promises “memory.” What they usually deliver is a vector store bolted onto a chat loop. That works for retrieval-augmented generation, but it’s not memory — it’s search.
Real memory needs to be:
- Selective — not everything is worth remembering
- Structured — raw text dumps don’t compose
- Decaying — old context should fade unless reinforced
The typical implementation stuffs every user message into a vector DB and calls it a day. When the agent retrieves context, it gets a grab bag of semantically similar fragments with no sense of recency, importance, or relationship between them.
The Three-Tier Pattern
The pattern that’s held up best in production separates memory into three distinct layers, each with its own write strategy and retrieval logic.
Working Memory
This is just the conversation context — what the agent is currently doing. Most frameworks handle this fine. The mistake is treating it as the only memory tier and trying to cram everything in here via prompt stuffing.
The key discipline: working memory should be curated, not accumulated. If your context window is filling up with “just in case” information, you’ve already lost.
Episodic Memory
Structured records of past interactions, decisions, and outcomes. Not raw transcripts — distilled summaries with metadata: what happened, when, what was the result, what was surprising.
The write trigger matters enormously. Writing after every exchange creates noise. Writing only on explicit “save” misses implicit learning. The sweet spot I’ve found: write on state transitions. When a task completes, when an error occurs, when the user corrects the agent, when a decision is made.
Each episode gets typed metadata — not just an embedding. You need to be able to query “what happened last time we deployed to production” without hoping the vector similarity gods are feeling generous.
Semantic Memory
Distilled knowledge and preferences that emerge from episodes. This is where “the user prefers concise responses” or “this codebase uses factory patterns” lives. It’s the agent’s long-term understanding of its environment.
Semantic memory should be written reluctantly and updated carefully. A single correction from the user is an episode. Three corrections about the same thing is a semantic memory. This prevents one-off preferences from ossifying into permanent behavior.
The Decay Problem
Memory without forgetting is hoarding. Old episodes should lose relevance over time unless they’re reinforced by new experiences. This isn’t just a storage optimization — it’s a correctness requirement. The user’s preferences from six months ago may be wrong today.
I use a simple reinforcement model: every time a memory is retrieved and not contradicted, its retention score gets a small bump. Memories that are never retrieved gradually decay. Memories that are retrieved and contradicted get flagged for review rather than silently updated.
What Doesn’t Work
Flat vector stores. They retrieve by semantic similarity, which is not the same as relevance. “How do I deploy” matches both your deployment guide and that time deployment broke at 3am. Without metadata filtering, you’re playing retrieval roulette.
Unlimited context windows as a substitute. Even with 1M+ token windows, dumping everything in the prompt is a bad strategy. The agent loses focus, latency increases, costs go up, and you’re still not actually learning across sessions.
LLM-generated summaries as memory. Summaries lose detail and inject hallucinated connections. Use them for display, not for retrieval. The source of truth should be structured records, not prose.
The Takeaway
Memory isn’t a feature you bolt on. It’s an architectural decision that affects how your agent reasons, what it can learn, and how it degrades over time. Get the tiers right, be disciplined about write triggers, and build in decay from day one. Your future self debugging a confused agent at 2am will thank you.