How to add memory to LLMs with context engineering

TL;DR

Context engineering builds a dynamic information package for each interaction to give LLMs a working memory. Sessions act as the one conversation container with an event log and a mutable state. Memory stores long term knowledge and user specific patterns. Together they enable stateful, personalized AI behaviors across multiple sessions.

Transcript

Welcome back to the deep dive. Today we're jumping straight into something really fascinating, uh, kind of complex, but super important in AI right now. We're talking about how you actually give a large language model a working brain, something like memory. And our main source for this is the day three white paper from that uh, 5day of AI agents in... Read More

Key Insights

• Context engineering actively aggregates data from long term memory, documents, tools, and immediate dialogue to form a tailored prompt for each turn.
• The core problem of context rot is addressed by dynamic history mutation and selective summarization to keep the signal strong within the context window.
• A session is a self contained workspace for one user, containing events and a separate state that tracks progress and tool usage.
• Mutability in session state allows efficient in place compaction, enabling scalable multi agent or MAS setups with cleaner histories.
• Two main history designs exist for multi agent systems: shared unified histories and separate individual histories, each with trade offs in context richness and autonomy.
• Privacy and performance requirements drive design choices such as PII reduction before storage and asynchronous background updates to memory.
• Memory, or ARID, stores static factual knowledge about the world, while memory stores dynamic user specific knowledge and processes, enabling personalized interactions.
• Advanced history management uses recursive summarization and token based pruning to keep history within limits without losing essential information.

Questions & Answers

Q: What is context engineering in simple terms and why is it needed for LLMs to remember things?

Context engineering is the process of assembling a full, relevant prompt for each interaction by pulling from long term memories, documents, tool outputs, and the current conversation. It is needed because LLMs are stateless by default, so memory and context must be rebuilt every turn to preserve continuity, personalization, and accurate reasoning.

Q: What constitutes a session in the context of this approach and how does it relate to memory?

A session is a container for one conversation, including a chronological events log and a separate state that tracks the current task, tools used, and progress. It serves as the immediate working memory for that interaction, while memory stores longer term, user specific information that can be referenced across sessions.

Q: How does the system manage context rot and why is asynchronous background processing important?

Context rot happens when the context window becomes too noisy or full, reducing the model’s ability to focus on important signals. Asynchronous background processing allows heavy tasks like recursive summarization to run without blocking the user, enabling ongoing compression of history and faster turn responses.

Q: What is the difference between a shared history and separate histories in multi agent systems?

Shared history lets all agents read and write to a common log, which supports tightly coordinated delegation but can become cluttered. Separate histories keep agents more autonomous and isolated, but require an abstract memory layer to share synthesized knowledge without leaking private context.

Q: Why is memory separated into declarative and procedural types, and what do they cover?

Declarative memory stores static facts and personal details, helping the agent know the world and the user. Procedural memory stores skills and workflows, capturing the knowledge of how to perform tasks and the exact sequence of tool calls needed for complex activities, enabling consistent performance across sessions.

Q: What security and privacy considerations are highlighted for session data?

The approach emphasizes strict isolation and privacy protections, including PII reduction before storage to comply with regulations, and data hygiene measures like TTL policies to govern how long data is kept. These practices are essential due to the hot path nature of session data and potential exposure risks.

Q: How does the system distinguish between ARID memory and user memory, and why are both needed?

ARID memory acts as a knowledge librarian for static facts and world knowledge, while user memory behaves like a personal assistant that captures dynamic, user specific information. Both are needed to provide accurate general knowledge and personalized, context aware responses that reflect the user’s history and preferences.

Q: What role do vector databases play in this architecture and what is their typical use case?

Vector databases are commonly used for fast, semantically aware search over memories and documents. They enable efficient retrieval of relevant knowledge and past interactions, which feeds into the context engineering pipeline to assemble precise prompts for the LLM on each turn.

Summary & Key Takeaways

• Context engineering is the foundation that feeds the LLM with the right data for every turn, including system instructions, tool definitions, and relevant retrieved documents.

• A session captures one conversation with a chronological log and a structured working memory, while memory stores long term, user specific knowledge to enable personalization over time.

• Memory and sessions work in tandem to decouple short term interaction history from long term knowledge, supporting both shared and separate histories among multiple agents.