Context Engineering and Agent Memory: What AI Should Know, Remember, and Retrieve
Oct 25, 2026
13 min read
2 views
Last updated: October 2026
One of the most important AI shifts in Silicon Valley is not another chatbot interface. It is the move from stateless assistants that answer one prompt at a time to agents that can remember, retrieve evidence, use tools, and continue work across hours, days, or months.
The name for the discipline behind that shift is context engineering.
Prompt engineering asks, "What should I say to the model?" Context engineering asks a larger question: "What should the model know, remember, retrieve, and be allowed to do at this moment?"
This article explains why the shift is happening, how agent memory differs from retrieval and long context, where these systems are useful, and how to build a practical context workflow without filling the model with unnecessary information.
Why context engineering is becoming the new AI stack
Large language models do not automatically know your current project, past decisions, preferred sources, or what you learned last month. A longer context window helps, but putting every conversation and document into every request is slow, expensive, noisy, and often counterproductive.
In September 2025, Anthropic described context engineering as the natural progression of prompt engineering. Its definition includes the instructions, tools, external data, conversation history, and other information selected for a model at inference time. Anthropic also highlighted structured note-taking as a form of agent memory that persists outside the context window.
By 2026, this idea had moved from engineering advice into major products:
Anthropic's Managed Agents architecture uses durable session history, compaction, and memory to support long-running work.
OpenAI's updated Agents SDK introduced configurable memory, sandbox-aware orchestration, and filesystem tools for agents that work across many steps.
OpenAI's new ChatGPT memory architecture focuses on synthesizing memories that remain relevant, current, and useful over long periods.
Google's Gemini Enterprise Agent Platform added Memory Bank, Memory Profiles, and persistent project context for agents and teams.
These systems differ in implementation, but their direction is the same. Better models still matter. The harder product problem is increasingly deciding what enters the model's limited attention, what stays outside it, and how the system retrieves the right information when it is needed.
Context engineering in plain English
An AI agent's context is the working material available for its next decision. It may contain several layers:
Instructions: Goals, rules, and boundaries, such as "Use primary sources and cite every factual claim."
Working state: What is happening in the current task, including the current draft, open questions, and completed steps.
Retrieved knowledge: Evidence selected for this request, such as three relevant papers and five saved highlights.
Durable memory: Information retained across sessions, including preferences, past decisions, and project history.
Tools and permissions: What the agent can access or change, such as search, files, calendars, or a read-only database connector.
Context engineering is the work of selecting, organizing, refreshing, and protecting those layers. A strong system sends enough context to make the next action correct without flooding the model with everything it has ever seen.
This is why the task is broader than writing a clever prompt. Even a perfect instruction cannot compensate for missing evidence, stale memory, the wrong tool, or an irrelevant document retrieved at the wrong time.
Context engineering, RAG, memory, and MCP are not the same thing
These terms are often grouped together, but they solve different parts of the problem.
RAG retrieves knowledge
Retrieval-Augmented Generation searches an external collection for information relevant to a request, then gives selected passages to the model. It is useful for large libraries that cannot fit into one prompt and for material that changes after the model was trained.
Memory preserves continuity
Memory carries information across turns or sessions. It might store an explicit preference, a project decision, a summary of earlier work, or a lesson learned from an agent's previous attempt.
MCP connects tools and data
The Model Context Protocol gives an AI client a standard way to connect to external services. MCP can expose search, files, databases, or actions, but the protocol does not decide which memory is true or which source should be retrieved.
Context engineering coordinates the whole system
Context engineering decides what to retrieve, which memories are relevant, how much history to include, what tools to expose, and what evidence the model must cite. RAG, memory, and MCP are components inside that larger design.
Why a larger context window does not solve the problem
It is tempting to treat context as a storage problem: if the model can accept more tokens, give it everything. That fails for several reasons.
Relevance
The model needs the right evidence, not the maximum amount of evidence. Old tool output, repeated summaries, and loosely related documents consume attention that should go to the decision at hand.
Freshness
A memory can be accurate when written and wrong six months later. OpenAI's 2026 memory work explicitly treats staying current as a separate evaluation target because time changes plans, locations, roles, and preferences.
Provenance
"The user believes X" is different from "the user highlighted an author who believes X." Without a source and a distinction between quotation, interpretation, and inference, an assistant can turn reading history into a false claim about the reader.
Permission
Personal context often includes private research, work material, medical information, or unfinished thinking. A useful memory system needs scope and access controls, not only retrieval accuracy.
Conflict
People change their minds. Sources disagree. A memory system must preserve time and provenance well enough to show conflict rather than silently merging it into one confident statement.
The engineering problem is therefore curation under constraints. The system must maximize useful, trustworthy context while minimizing noise, staleness, cost, and risk.
How agent memory actually works
Memory is not one database that an agent reads before every answer. A useful system performs a cycle of decisions.
1. Decide what is worth remembering
The system first identifies information that may matter later. Explicit statements such as "Use metric units" are strong candidates. Temporary details such as "I am waiting at gate 12" should usually expire. An inference about a person's politics or health should face a much higher bar than either.
2. Store the information with metadata
A memory needs more than text. Useful metadata may include its source, owner, creation time, confidence, privacy scope, and whether it was stated by the user or inferred by the system.
3. Retrieve only what fits the current task
At answer time, the system searches memory and external knowledge for a small set of relevant items. Retrieval should consider topic, recency, authority, and permission rather than semantic similarity alone.
4. Reconcile conflicts and changes
New information may update an old preference or contradict a previous source. A robust system preserves enough history to recognize the change instead of combining both into a misleading answer.
5. Forget, archive, or ask
Some memories should decay, some should remain available but inactive, and some should be confirmed with the user. Deletion and correction are core memory operations, not administrative extras.
This cycle explains why memory is harder than storing chat history. The difficult work lies in deciding what remains useful and trustworthy over time.
A practical context engineering playbook
You do not need to build an agent platform to use these principles. They improve ordinary work with ChatGPT, Claude, coding agents, and research assistants.
Step 1: Define the task state
State the goal, current position, completed work, open questions, and definition of done. For a long task, keep this state in a short external note that can be updated between sessions.
Step 2: Separate context by role
Keep instructions, evidence, memory, and working notes distinct. A model should be able to tell the difference between a rule it must follow, a source it may quote, a tentative hypothesis, and an old decision that can be reconsidered.
Step 3: Retrieve narrowly
Ask for the smallest useful evidence set. For example:
Find up to eight primary-source passages about agent memory that directly address freshness, provenance, or permissions. Exclude general AI commentary.
This produces a better working set than "Tell me everything about agent memory."
Step 4: Preserve sources and uncertainty
Attach a source to factual claims and label inferences. If a memory came from a model-generated summary, preserve the original material behind it. If the sources disagree, show the disagreement.
Step 5: Compact without destroying recoverability
Summarize old work to save context, but keep the underlying record accessible. Anthropic's Managed Agents architecture makes the same distinction: the active context can be transformed while durable session events remain available for later inspection.
Step 6: Evaluate the trajectory, not only the final answer
Check whether the agent selected the right sources, used the appropriate tools, respected permissions, and recovered from errors. A plausible final paragraph can hide a poor or unsafe process.
Where context engineering is useful now
Research
A research agent can keep the question and inclusion criteria in working state, retrieve evidence from a larger collection, record which sources were rejected, and generate a cited synthesis. The benefit is not merely faster summarization. It is continuity across a multi-step investigation.
Software development
A coding agent needs repository instructions, architecture decisions, current test failures, and progress from earlier sessions. Persistent project notes help a new session continue from the real state of the work instead of restarting from the original request.
Team operations
An internal agent can retrieve policy, customer history, and past decisions while respecting role-based permissions. Durable context can reduce repeated explanations, but only if the organization knows which system is authoritative and when information was last updated.
Personal learning and creative work
A personal assistant can retrieve reading notes, unfinished ideas, prior drafts, and questions the user has revisited over time. Here, the most useful memory is often not a complete conversation log. It is a smaller record of what the person selected, interpreted, and chose to keep.
The risks every context system should address
More memory is not always better
An agent that remembers everything can become slower, more expensive, and less accurate. Retention should be selective, reversible, and visible.
Personal archives can reinforce a filter bubble
People save what interests or confirms them. Retrieval should sometimes surface contradictions, missing perspectives, or the original surrounding context.
A source can be wrong
Provenance makes an answer auditable, not automatically true. The system should preserve the difference between a sourced claim and a verified claim.
Private context requires strong boundaries
Personal learning histories can reveal beliefs, health concerns, career plans, and relationships. Users need clear control over what is private, what is shared, and what each connected assistant can access.
Agent memory can turn inference into identity
Saving an article about a topic does not mean the user agrees with it. Memory should favor explicit user statements and source-linked notes over personality claims inferred from behavior.
What this means for personal knowledge tools
Most AI memory systems begin with conversation history. Personal knowledge tools begin with a different signal: what a person deliberately selected while reading, watching, and thinking.
That selection is useful. A full article says what an author published. A highlight says which part mattered to the reader. A note adds why it mattered. A tag adds a topic. A timestamp or URL preserves where the idea came from.
Together, these elements form a compact unit of human-curated context:
source + selected passage + personal interpretation + topic + time
This is richer than an isolated bookmark and easier to audit than an AI-generated memory with no visible origin. A personal knowledge tool can therefore act as a context layer without trying to replace the agent or the model.
How this applies to Glasp
Glasp is one practical example of this pattern. It already captures selected material from web articles, PDFs, YouTube transcripts, and Kindle highlights. A person can keep the exact source passage, add a note, and return to its original context.
Its MCP Connector lets compatible AI assistants search a user's highlights and memories in natural language. The connector is scoped to the user's account. It reads highlights, notes, summaries, and bookmarks; the only thing it can add is a bookmark, when the user asks for one. It cannot post, edit, or delete highlights.
Glasp's AI Clone uses the same underlying idea inside Glasp. It answers from a person's highlights and notes and exposes the contributing sources for inspection.
This creates several immediate uses:
A writer can retrieve saved passages on one topic and draft from cited evidence.
A learner can ask an assistant to quiz them using their highlights.
A researcher can resurface a passage from a PDF, Kindle book, article, or YouTube transcript without remembering where it appeared.
A user can carry the same knowledge library into Claude, ChatGPT, Claude Code, Cursor, or another compatible client instead of rebuilding context in every conversation.
A practical request might look like this:
Search my Glasp highlights for up to eight passages about AI memory. Group them into benefits, risks, and open questions. Distinguish the author's claim from my notes, and include the original source for every passage.
The next request can generate an output from that small evidence set:
Using only those passages, draft a 500-word argument. Label any inference that is not directly supported by a source.
The final step remains human verification. The user should open the cited sources, check whether the passages support the claims, and update any note whose interpretation has changed.
Why this shift matters
Models are becoming more capable and more interchangeable for many everyday tasks. Personal context is not interchangeable. It accumulates through what a person reads, questions, keeps, rejects, and changes their mind about.
That makes the next generation of knowledge tools different from traditional note apps. Their job is not only to store information for a person to find later. Their job is to help the person decide what an AI should know now, what it may remember later, and what evidence must remain attached.
Glasp is well positioned because highlighting is already an act of context selection. The next step is to make that context easier to scope, verify, refresh, and carry across agents.
Frequently asked questions
What is context engineering?
Context engineering is the practice of selecting and managing the instructions, working state, retrieved knowledge, memory, tools, and permissions an AI model receives for a task. It is broader than prompt engineering because it covers the whole information environment around the prompt.
How is context engineering different from prompt engineering?
Prompt engineering focuses on how an instruction is written. Context engineering also decides what evidence, history, memories, tools, and constraints accompany that instruction. A good prompt with the wrong context can still produce a poor answer.
Is context engineering the same as RAG?
No. RAG is one technique for retrieving relevant external knowledge. Context engineering also includes memory, conversation state, tool selection, permissions, compaction, and decisions about what not to include.
What is agent memory?
Agent memory is information kept outside a single model response and reused later. It can include user preferences, previous decisions, project state, summaries of earlier work, or lessons an agent learned from a past attempt.
Why can an AI not simply remember everything?
Unlimited retention creates noise, cost, privacy risk, stale information, and contradictions. Useful memory requires selection, freshness checks, provenance, and user control.
How does MCP relate to context engineering?
MCP is a standard connection layer that lets AI clients access external tools and data. It can carry Glasp highlights into an assistant, while context engineering determines when those highlights should be retrieved and how they should be used.
How can I use Glasp as AI memory today?
Save focused highlights and notes, connect Glasp to a compatible assistant through the MCP Connector, and ask for a small set of relevant passages with sources. Use those passages to write, study, or make a decision, then verify the original sources.
Can a connected AI change my Glasp data?
Mostly no. The Glasp MCP Connector searches and reads the user's own highlights, notes, summaries, and bookmarks. It cannot create, edit, or delete highlights. The one thing it can add is a bookmark, and only when the user asks it to save a page.
Primary sources
Effective Context Engineering for AI Agents, Anthropic, September 2025.
Effective Harnesses for Long-Running Agents, Anthropic, November 2025.
Scaling Managed Agents: Decoupling the Brain from the Hands, Anthropic, April 2026.
The Next Evolution of the Agents SDK, OpenAI, April 2026.
Dreaming: Better Memory for a More Helpful ChatGPT, OpenAI, June 2026.
The New Gemini Enterprise: One Platform for Agent Development, Orchestration, and Governance, Google Cloud, April 2026.
Announcing Vertex AI Agent Engine Memory Bank, Google Cloud, July 2025.
Before you leave
The era of the isolated prompt is ending. Useful AI increasingly depends on a carefully managed relationship between model, memory, evidence, tools, and permission.
The practical starting point is small: save fewer but better passages, attach your interpretation, preserve the source, and retrieve only what the current task needs. That is context engineering at a personal scale.