Think about your own memory. It isn’t a single, monolithic thing. You have short-term memory that holds what someone just said to you, and you have long-term memory that stores foundational experiences from your childhood. To be truly intelligent and useful, an AI agent needs both.
We’ve previously discussed context windows—the finite amount of information an LLM can “see” at one time. This is the agent’s short-term memory. It’s fast and effective, but what happens when a conversation gets too long and early details fall out of the window? How can an agent remember a user’s preference from a conversation last week?
To solve this, we must equip our agents with different types of memory architectures. Let’s explore the spectrum, from simple conversational buffers to sophisticated, persistent vector stores.
Part 1: Short-Term Memory (Managing the Current Conversation)
Short-term memory techniques are designed to manage information within the constraints of a single interaction and the model’s context window.
1. Conversation Buffer Memory
This is the most basic and intuitive form of memory. It simply stores the entire conversation history verbatim and stuffs it into the prompt on every turn.
- Concept: Keep a raw log of all user and AI messages.
- Analogy: A court stenographer recording every single word said during a trial.
- Pros: Perfect, lossless recall of the immediate conversation. The agent sees every detail exactly as it happened.
- Cons: Extremely token-intensive. It’s the most expensive and least scalable method. In long conversations, it will quickly exceed the context window limit, leading to errors.
- Best For: Short, task-oriented conversations where every single word is critical.
2. Conversation Summary Memory
To overcome the limitations of the buffer, this method uses an LLM to periodically create a running summary of the conversation.
- Concept: As the conversation grows, an LLM condenses the history into a concise summary. This summary is then passed into the prompt instead of the full transcript.
- Analogy: Instead of the full trial transcript, the agent is given the running “meeting minutes” that summarize the key points.
- Pros: Far more token-efficient than a simple buffer, allowing for much longer conversations.
- Cons: Can lead to information loss. The LLM might decide a detail isn’t important enough for the summary, but it becomes critical later. It also incurs the cost and latency of additional LLM calls to perform the summarization.
- Best For: General-purpose chatbots where the overall gist and key entities of the conversation are more important than the exact phrasing of every message.
3. Token Window Buffer Memory
This is a “sliding window” approach that only keeps the most recent portion of the conversation.
- Concept: The memory is constrained to the last
knumber of tokens or conversation turns. - Analogy: A person who can only remember the last five minutes of a conversation.
- Pros: Simple to implement and has a predictable, fixed token cost.
- Cons: Abruptly forgets anything that happened early in the conversation, regardless of its importance.
- Best For: Applications where only the most recent context is relevant, like a live customer support agent that doesn’t need to know about a ticket from an hour ago.
Part 2: Long-Term Memory (Recalling Across Sessions)
Short-term memory is wiped clean after each session. To build an agent that can learn and personalize its interactions over time, we need a mechanism for long-term memory that persists indefinitely.
The challenge is clear: you can’t just load a user’s entire multi-year history into a prompt. The solution is to retrieve only the most relevant memories for the current task. This is the superpower of Vector Stores.
- Concept:
- Storage: After an interaction, key pieces of information (like user preferences, facts, or conversation summaries) are converted into numerical representations called embeddings and stored in a vector database.
- Retrieval: When a new conversation starts, the user’s current message is also converted into an embedding.
- Search: The agent performs a “semantic search” in the vector database. It doesn’t look for keywords; it looks for past memories that are conceptually similar to the current message.
- Injection: Only the top few, most relevant memories are retrieved and inserted into the agent’s prompt, providing crucial long-term context without overwhelming the context window.
- Analogy: It’s like having a magical research assistant. You say, “Remind me about my goals for the marketing project we discussed.” The assistant doesn’t read you your entire work diary. Instead, they instantly find the three specific entries from last month that are most conceptually related to “goals” and “marketing project” and present them to you.
- Pros: Infinitely scalable long-term memory. Highly token-efficient. Enables true personalization and learning over time.
- Cons: More complex to set up and manage. The quality of recall depends heavily on the quality of the embedding model and the retrieval strategy.
Summary Table
| Memory Type | How it Works | Pros | Cons | Best For… |
| Conversation Buffer | Stores full conversation history. | Perfect recall. | High token cost, not scalable. | Short, detail-oriented tasks. |
| Conversation Summary | LLM creates a running summary. | Token-efficient for long chats. | Potential info loss, extra cost. | General-purpose chatbots. |
| Token Window Buffer | Keeps only the last k turns/tokens. | Predictable cost, simple. | Forgets old but important info. | Applications needing recent context only. |
| Vector Store | Stores embeddings, retrieves by similarity. | Scalable, personalized, efficient. | Complex setup, dependent on quality. | Agents needing to learn over time. |
Conclusion
Memory is what elevates an agent from a stateless tool to a stateful, personalized partner. The most sophisticated agents often use a hybrid approach: a short-term buffer for perfect immediate recall, combined with a vector store for retrieving relevant long-term knowledge. By understanding these different architectures, you can design agents that not only solve problems but also learn, adapt, and build lasting context with their users.
Author
Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins. In my free time, I write blogs on ckdbtech.com