Context Rot in LLMs: RAG vs Long Context in 2026

What Context Rot Is in LLMs

Context rot is the measured drop in a large language model's accuracy as its input grows, well before the context window is full. It comes from attention spread across more tokens, distractors that look like the answer, and training that rarely rewards reasoning at extreme lengths. The fix is to send less: retrieve and rerank, clear stale tool output, compact or restart long sessions, and measure your own safe budget instead of trusting the spec sheet.

Chroma Research put numbers on it across 18 models in July 2025, and the term has gone mainstream enough that Anthropic's own documentation now defines it: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot." A bigger advertised window raises the ceiling on what you can send. It doesn't promise the same accuracy at the top of that range as at the bottom.

The practical version, for anyone deciding what to build: the documented context window is what you're allowed to send. Your usable window is where the model still hits your quality bar. Those are different numbers, and the second is usually a fraction of the first.

That gap is why the argument that ran through engineering channels in late 2024 and early 2025 ("RAG is obsolete now, just paste everything in") aged badly. The windows did arrive. Anthropic makes 1M tokens the default on Claude Opus 5.5, Sonnet 5.5, and the new Haiku 5.5. OpenAI's GPT-6 models carry a 1.05M window, of which 922K can be input. Google's Gemini 3.1 Pro and Gemini 3.8 Flash accept about 1M input tokens. Meta's Llama 4 Scout documents 10M. What didn't arrive was a model that uses every part of its window equally well.

So the interesting question stopped being "which one wins" and became "which pattern fits this data shape, this latency budget, this freshness requirement, and this bill." That's what the rest of this piece answers, starting with what the research measured and how to fix it.

---

What Chroma's Context Rot Research Showed

The headline ("context rot exists") undersells the work. In Context Rot: How Increasing Input Tokens Impacts LLM Performance, a technical report published July 14, 2025, and the study most people mean by "the context rot paper," Kelly Hong, Anton Troynikov, and Jeff Huber ran controlled experiments across 18 models: Claude Opus 4, Sonnet 4, Sonnet 3.7, Sonnet 3.5, and Haiku 3.5; o3, GPT-4.1 with its mini and nano versions, GPT-4o, GPT-4 Turbo, and GPT-3.5 Turbo; Gemini 2.5 Pro, 2.5 Flash, and 2.0 Flash; and three sizes of Qwen3. Not every model ran in every experiment, because of window and thinking-budget limits. Their replication kit on GitHub lets you rerun it.

Four findings matter for architecture.

Performance degrades unevenly as input grows. If you came here looking for the context rot graph, the shape is the point: there isn't one curve. Chroma's own summary is that models "do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows." How fast, and where, depends on the model and the task, which is why a spot check at one context size tells you almost nothing about behavior at another. Anthropic's engineering team describes the same pattern as "a performance gradient rather than a hard cliff."

The size of the drop is large. On LongMemEval, a conversational question-answering benchmark, the team filtered down to 306 prompts and compared two versions of each. The focused version contains only the relevant material and averages about 300 tokens. The full version buries the same answer in the surrounding conversation and averages about 113K tokens. Performance degraded consistently on the full versions. The Claude models showed the widest gap, and Chroma attributes much of it to abstention: Claude Opus 4 and Sonnet 4 often declined to answer rather than guess. Same question, same model, same answer sitting in the input. The main change was how much irrelevant text came along.

Similarity shapes the decline. When the needle is less semantically similar to the question (Chroma scored this with embedding similarity), accuracy falls faster as the input grows. Distractors that resemble the answer hurt too: even a single one reduces performance against the needle-only baseline, and four compound it. Position, by contrast, made little difference in this task. Chroma saw no notable variation across 11 needle positions, even though Liu et al. (2024), "Lost in the Middle," in TACL found a U-shaped curve on multi-document question answering, where the middle of a long context is systematically underweighted.

Coherent filler hurt more than shuffled filler. This is the result that should change how engineers think. The intuition says a coherent haystack should be easier to search than a jumbled one. Chroma found the opposite, consistently, across all 18 models: shuffling the haystack's sentences and destroying its logical flow improved performance. Why that happens is still open, and Chroma only goes as far as saying the structure of an input could influence how the attention mechanism is applied. The actionable part needs no mechanism: coherent, on-topic padding is not automatically harmless.

One more detail to carry into production: failure modes are model-specific. Under distractors, Claude models showed the lowest hallucination rates, with Claude Sonnet 4 and Opus 4 especially conservative and inclined to abstain when uncertain, while GPT models showed the highest and tended to answer confidently and wrongly. If you're choosing between them, you're partly choosing which failure you'd rather debug.

Chroma isn't the only team measuring this. If you're looking for a context rot benchmark that resembles your workload, these are the main ones:

That last row is the one to remember: even when a model retrieves every relevant passage perfectly, a longer input still costs accuracy.

For a gentler entry point, Hamel Husain hosted Kelly Hong to walk through the research and published an annotated write-up of that talk. Its takeaways land where the report does: performance isn't uniform across context lengths, how you present information matters, and deliberate context engineering is the lever.

---

Why Long Context Fails: The Mechanism

The mechanism matters because it predicts which workloads break.

Transformer attention runs a softmax over token pairs. As sequence length grows, attention spreads across more positions. Even with relative position encodings like RoPE or ALiBi, the softmax denominator grows and the weight available for any single token shrinks. At 1M tokens, the right token competes with 999,999 others for a finite budget. Anthropic's engineers call this an "attention budget": "Every new token introduced depletes this budget by some amount."

Position encodings stretch the range without removing the problem. RoPE degrades when extrapolating far past training length, so a model trained on 32K sequences and deployed at 1M is doing extrapolation the underlying math doesn't fully support. Meta, for example, pre-trained and post-trained Llama 4 Scout at 256K and documents a 10M window, so everything past 256K leans on length generalization. YaRN, position interpolation, and NTK-aware scaling all help, and none of them produce a model that uses 1M tokens as well as it uses 32K.

There's a training data problem too. Even when a model trains on long sequences, examples requiring genuine reasoning across 800K tokens are rare. Models learn to use the parts of context their training data taught them to use.

So context rot is a property of the architecture and the training distribution rather than a bug the next release patches out. Future models will push the frontier further, and the shape of the degradation will persist.

---

Premature Termination: What 2026 Research Added

The most useful new result since this article was first published came out of long-horizon agent work rather than benchmark work.

In Diagnosing and Mitigating Context Rot in Long-horizon Search, first posted in June 2026 and substantially revised in August, Shijie Xia, Yikun Wang, Zhen Huang, and Pengfei Liu studied four open-weight flagship models (GLM-4.7, GLM-5.0, Qwen3.5-397B-A17B, and MiniMax-M2.5) on three deep-search benchmarks: BrowseComp, xbench-DeepSearch, and BrowseComp-Plus, which searches a fixed local corpus. Closed models were left out because their reasoning traces are encrypted. The revised version names a failure mode the earlier literature had skipped: premature termination. Under a large context, models give up, or hand back a wrong answer they flag as uncertain, long before they've exhausted the window. Controlling for how hard the query was, the authors found the premature termination rate rises with context length.

That reframes the problem. Context rot covers two different failures, and they need different fixes. If the model can't find the needle anymore, improve precision. If the model stops looking, it needs room and reason to keep exploring. For an agent doing multi-step search, conflating them wastes weeks.

Their second finding is the one to steal. They tested seven context-management methods in three families (compaction, trimming, and isolation through sub-agents) and concluded these methods work mainly as test-time scaling strategies: by cutting the premature termination rate, they buy the model more exploration. Which family helps depends on the model. On the stronger agentic model they compared, Qwen3.5-397B-A17B, isolating work in sub-agents scored best. On the weaker one, GLM-4.7, sub-agents were among the worst options, and keeping only the latest tool results plus a summary gave the best balance of accuracy and cost. The authors also built a behavior-aware filtering strategy for parallel sampling that raised average accuracy by 2.6 to 4.9 percentage points across three aggregation methods.

If you run agents over long sessions, track how often yours quit early. It's a cheap metric and almost nobody instruments it.

Three more 2026 preprints, posted between July and early October, push in the same direction. Context as an Environment keeps tool outputs as variables in a persistent Python session and lets only what the agent prints enter its context; with the same Qwen3.8-Max model underneath, it scored 86.7 percent on the LOCA benchmark at 256K tokens, against 65.3 percent for an agent that summarizes its history. Stateless Language Agents rebuilds the context on every call from state the harness holds, instead of carrying the conversation forward, and on kernel optimization reached the strongest baseline's final performance with over 84 percent fewer tokens. And CompactionRL treats compaction as a skill to train rather than a feature to bolt on: its GLM-4.5-Air model scored 6.6 points higher on SWE-bench Verified than the same base model with compaction added at inference time. These are preprints, but they agree on the direction: keep the working set small, and make the model good at keeping it small.

---

How to Fix Context Rot in Agents and Chat Tools

Every fix that works does the same thing: it puts fewer, better tokens in front of the model. Anthropic's engineering team says it in one line in Effective context engineering for AI agents: "Context, therefore, must be treated as a finite resource with diminishing marginal returns." What changes is where you apply the rule.

If you build LLM apps or agents:

• Send less, and choose it well. Retrieve, rerank, and drop what the task doesn't need. Google's long-context guide says the same: "Generally, if you don't need tokens to be passed to the model, it is best to avoid passing them."
• Put long documents first and the question last. For inputs of 20K tokens or more, Anthropic's prompting guide recommends documents at the top and the query at the end, noting that queries at the end "can improve response quality by up to 30 percent in tests." It also suggests asking the model to quote the relevant passages before it answers. Google gives the same placement advice for Gemini.
• Clear stale tool results. Old search results and file dumps are the cheapest tokens to delete. Anthropic calls tool-result clearing "one of the safest lightest touch forms of compaction" and ships it as context editing, which by default starts clearing at 100,000 input tokens and keeps the three most recent tool uses.
• Compact before the window fills, and test the summary. Anthropic's API offers threshold compaction (default trigger: 150,000 input tokens) and, since September 2026, on-demand compaction. OpenAI's Responses API compacts server-side when you set a compact_threshold under context_management, and also exposes a standalone compact endpoint. A summary is lossy, so check what it drops. Anthropic's advice is to start by maximizing recall, then cut what's superfluous.
• Keep rules and state outside the summary. The Compaction Cliff in Long-Running AI Agent Memory, a CIKM 2026 paper posted to arXiv in August, found that Claude Code's compact prompt on Sonnet 4.6 kept 53 percent of the safety rules in production agent configurations after one round and 10 percent after five. The authors' fix pins safety rules and procedures verbatim and compresses the rest by type, and it held 96 percent recall over five rounds. So pin rules where the agent rereads them, and keep progress in notes, a memory tool, or git rather than in chat history.
• Push exploration into sub-agents. A sub-agent spends its own context on the search and hands back "a condensed, distilled summary of its work (often 1,000-2,000 tokens)," in Anthropic's words. Test it on your model, though: in the premature termination paper, sub-agents scored best on the stronger of the two models compared and among the worst on the weaker one. We cover how sub-agents, skills, and hooks fit together in our agentic engineering guide.
• Measure by context size. Score your own eval at several input lengths and log early quits. Without that, every fix above is a guess.

Anthropic has published numbers for the combination. In an internal agentic-search evaluation, its memory tool plus context editing improved performance 39 percent over baseline, and context editing alone 29 percent. In a 100-turn web search test, context editing cut token use by 84 percent. Those are a vendor's figures about its own product, so read them as a direction rather than a promise.

If you use chat assistants or coding agents:

• One chat per task. Claude Code's best-practices guide says: "A clean session with a better prompt almost always outperforms a long session with accumulated corrections." OpenAI's Codex guide agrees: "Keep one chat per coherent unit of work."
• Restart after two failed corrections. That's Claude Code's own rule of thumb: run /clear and write a better first prompt that includes what you learned.
• Compact on your terms. Claude Code, Codex, and Cursor all summarize automatically near the limit. Compacting earlier with a focus instruction (for example, /compact Focus on the API changes) gives you a say in what survives. Claude Code warns that "detailed instructions from early in the conversation may be lost," so persistent rules belong in a file the agent rereads, like CLAUDE.md, not in the chat.
• Watch the meter. /context in Claude Code shows what's filling the window, and /status in Codex shows how much of it you've used. Cursor puts a context ring next to the input box that opens a breakdown when you click it.

Commands and defaults change between releases, so treat the table as accurate for October 2026 and check your tool's docs after an upgrade.

---

Where RAG Still Wins

Given all that, retrieval-augmented generation still earns its place. Here's where it keeps winning.

Multi-document corpora at scale. If your knowledge base is 50,000 documents totaling 500M tokens, it fits in no window. Retrieval is the only viable architecture.

Freshness and recency. Vector stores update incrementally. A long-context prompt has to be rebuilt whenever content changes. For anything updating hourly (news, catalogs, support tickets, code), retrieval handles change cheaply.

Cost. Input cost scales with input tokens, and above certain thresholds it scales worse than linearly. If 95 percent of your queries can be answered from 5K relevant tokens, retrieval is a large multiple cheaper with no accuracy loss.

Citation and provenance. Retrieval hands you a structured list of sources you can show, link, and rank, while grounding a long-context answer in specific sources takes extra plumbing. This is the same reason a reading tool that keeps the passage and its source beats one that keeps a summary: when you ask questions across everything you've saved in Glasp, every answer can point back at the source it came from.

Access control and tenancy. If your corpus has per-user, per-tenant, or per-role visibility, you can't dump it all in. Retrieval filters by policy before the model sees anything. That's non-negotiable for B2B.

Multi-corpus reasoning. When the answer spans a Slack thread, a Notion page, a Linear issue, and a GitHub PR, retrieval is the bridge.

Check any of those boxes and RAG isn't optional. The question becomes how to make retrieval good, not whether to do it.

---

Where Long Context Wins

Long context has workloads where it's simply the right answer.

Single-document deep reasoning. Reading a 100-page contract and answering across clauses. Analyzing a paper. Working through an earnings call. When the answer connects two paragraphs 80 pages apart, chunking often severs the link.

Code understanding inside a repository. Plenty of code tasks need imports, types, definitions, and call sites at once. Chunking by file loses the relationships between files.

Conversational continuity. Long agent sessions benefit from real history. Retrieval over conversation history is brittle, because you usually need the last 50 turns, not the 50 most semantically similar ones.

Exploratory reasoning where you don't know the query yet. If you can't write the query in advance, retrieval is hard to aim. Long context lets the model browse.

Cross-reference within a coherent unit. A textbook chapter, a paper, a legal brief. Chunking and reassembling these tends to lose the argument.

Rough heuristic: if your data is one logical document and it fits inside your measured safe budget, long context is the cleaner architecture.

---

The Hybrid Pattern Most Teams Land On

The 2026 default for serious systems is neither pure RAG nor pure long context. Retrieve a substantial but bounded set of tokens, then reason over them.

User query
   |
   v
[Retrieval Stage]
   - Vector search (top 100 chunks)
   - Optional keyword/BM25 search merged in (hybrid retrieval)
   - Optional reranker (cross-encoder over top 100, keep top 30)
   |
   v
[Assembly Stage]
   - Concatenate retrieved chunks
   - Add metadata, source headers, structural hints
   - Target total: 50K to 200K tokens
   |
   v
[Long-Context Reasoning Stage]
   - Send to frontier model with reasoning prompt
   - Model uses the full retrieval set as its context
   |
   v
Answer + citations

Each stage covers the other's failure mode. Retrieval narrows a corpus that's too big for any window down to something manageable. Reasoning over the whole retrieved set restores the cross-chunk reasoning that classic top-5 RAG throws away.

The load-bearing decision is retrieval-set size. Too few tokens and you've just rebuilt top-5 RAG. Too many and you're in the rot zone, paying more for a worse answer. A workable rule of thumb, which the next section turns into a measurement: plan for a safe budget around 20 to 40 percent of the documented window, then verify it on your own data. For a 200K-window model that's 40K to 80K. For a 1M-window model it's 200K to 400K, which, as it happens, is right about where the billing changes at OpenAI and on Gemini 3.1 Pro.

---

Tuning the Hybrid: Numbers and Heuristics

These aren't universal truths. They're starting points that hold up in production.

Chunk size. 500 to 1,500 tokens for prose. 200 to 500 for code, per function or logical block. 1,500 to 3,000 for legal or academic text where within-chunk context carries meaning. Overlap by 10 to 20 percent.

Top-k retrieval. Pull more than you'll send. Retrieve top 50 to 200, then rerank. A cross-encoder costs more per pair than an embedding model and is dramatically better at fine-grained relevance.

Rerank-to-context ratio. Keep the top 20 to 100 chunks after reranking. The exact number follows from chunk size and your safe budget.

Hybrid retrieval. Combine dense and sparse (BM25, SPLADE) with reciprocal rank fusion. Dense alone misses exact matches like SKUs, error codes, and proper nouns. Sparse alone misses paraphrases.

Two of these dials deserve more than a bullet, because they're where teams lose accuracy without noticing.

The first is your safe-context budget, and the only honest way to set it is to measure. Build a small eval set of questions that need reasoning across several chunks, then score accuracy at 16K, 32K, 64K, 128K, and 256K of stuffed context. Take the largest size that still clears your bar and run 20 percent under it for headroom. That measured number should land somewhere near the 20 to 40 percent rule above, and when it doesn't, trust your eval over the heuristic. If you don't have an eval harness yet, our guide to the LLM evals stack covers the tooling.

The second is what happens when you upgrade the model, which bit teams in 2026. Anthropic's tokenizer changed with Claude Opus 4.7, and the same text now maps to roughly 1.0 to 1.35 times as many tokens depending on the content, which Anthropic summarizes as about 30 percent more. Haiku 5.5, released October 7, uses the newer tokenizer too, so a pipeline moving up from Haiku 4.5 counts roughly 30 percent more tokens for the same text. Your documents didn't grow. Your budget shrank. If you pinned a token ceiling a year ago and swapped the model underneath it, that ceiling means something different now, and nothing in your logs will announce it.

Bypass retrieval entirely when the query says so. "Summarize the document I just uploaded" is a single-document task. Detect those with a small classifier, skip retrieval, save latency, and avoid surfacing unrelated noise.

Summarization and pruning layers. For very long histories, compress older material before assembly. Summaries cost tokens too, so measure whether they actually help.

---

Anti-Patterns Engineers Keep Shipping

A few traps recur often enough to name.

"Just dump everything in context." Tempting after every release that doubles the window. It degrades silently, so you pass spot checks and fail in production on exactly the queries that needed cross-context reasoning. Run an eval at your target size before shipping.

"Always use RAG." Reflexive retrieval misses single-document cases. Indexing a 50-page PDF and pulling top-5 chunks usually beats nothing and loses to just sending the PDF.

"Ignore the assembled token count." Teams set top-k to "whatever fits," then discover three months later that their average prompt is 350K tokens, accuracy has quietly slipped, and the bill doubled at a threshold nobody read. Track assembled context size as a first-class metric and alert on it.

"Let auto-compaction handle it." Automatic compaction keeps a session alive. It doesn't keep it accurate. Summaries drop instructions, and the Compaction Cliff paper watched rule retention fall round after round. Decide what must survive, and store it outside the conversation.

"Skip evals because the model is good now." Model upgrades change the right architecture, and sometimes change your token accounting underneath you. Re-evaluate when models change.

"Assume a quiet failure is a safe failure." Under distractors, some models hallucinate and some abstain, and in long-horizon work some just stop early. Know which one yours does, and instrument for it.

---

What Changed in 2026: Models, Prices, and Compaction

Most of what follows changed between September and early October 2026, when Anthropic, OpenAI, and Google all shipped new models, and not all of it points the same way.

The model lineup turned over. Anthropic released Claude Fable 5.1 on September 1, Opus 5.5 on September 22, Sonnet 5.5 on September 28, and Haiku 5.5 on October 7, all with 1M-token windows. OpenAI shipped GPT-6 Astra on September 3, GPT-6 Sol and Luna on September 22, and GPT-6.1 Sol on September 29, each with a 1.05M window and up to 922K tokens of input. Google made Gemini 3.8 Flash generally available on September 2, while Gemini 3.1 Pro is still a preview model.

The best retrieval scores jumped, and the spread between models is huge. On OpenAI MRCR v2, its own 8-needle test, OpenAI reports GPT-6 Astra at 96.3 percent between 512K and 1M tokens, against 73.8 percent for GPT-5.6 Sol. Google's model card scores Gemini 3.1 Pro on Google's own MRCR v2 (8-needle), a different dataset despite the shared name: 84.9 percent averaged up to 128K and 26.3 percent at 1M. So the gap between vendors is a signal, not a head-to-head result. Both sets of numbers are vendor-run (OpenAI reports the best score at any reasoning effort), and MRCR measures finding and reproducing the right passage, not reasoning across a document. Context rot didn't go away. It became something you have to measure model by model.

Price cliffs now differ at every vendor. OpenAI still reprices the full request above 272K input tokens, and on the GPT-6 models that means "2x input and cache rates and 1.5x output." GPT-6.1 Sol goes from $2 and $10 per million input and output tokens to $4 and $15, and GPT-6 Astra from $10 and $50 to $20 and $75. GPT-5.6 Sol's $4 and $20 is a promotional price that runs at least through November 21, 2026, and above the same 272K line it rises to $8 and $30. Google bills Gemini 3.1 Pro prompts over 200K at $4 and $18 instead of $2 and $12, but Gemini 3.8 Flash has no such tier. Anthropic bills the full 1M window at standard rates on Claude 4.6 and later ("A 900k-token request is billed at the same per-token rate as a 9k-token request," per its pricing page), with one exception: on Haiku 5.5, prompts over 100,000 tokens cost five times as much, $0.50 and $2.50 instead of $0.10 and $0.50.

Compaction became a product. The settings are covered in the fixes above. What changed is that vendors now run compaction for you. Anthropic's docs recommend its new on-demand compaction "wherever it is available." OpenAI's Responses API returns compacted context as an opaque item that isn't meant to be human-readable, and its Agents API, in public beta since September 10, says "OpenAI handles session orchestration, context compaction, and recovery." Notice the default, though: a vendor with a million-token window starts threshold compaction at 150K.

Caching got cheaper. It still doesn't fix rot. Cache reads cost a tenth of the base input price on most Anthropic models, a twentieth on Opus 5.5 and Sonnet 5.5, and a fortieth on Fable 5.1. OpenAI charges a tenth on most models and a twentieth on GPT-6.1 Sol, and since GPT-5.6 it bills cache writes at 1.25 times the input rate. Caching makes repeated queries against stable content much cheaper. It doesn't change what the model does with a long input, and on OpenAI, cached tokens double in price past 272K, just as uncached input does.

What hasn't changed is the underlying result. Better models move the curve, a bigger window on its own fixes nothing, and you still need retrieval for fresh, multi-tenant, multi-source workloads.

This is the same discipline that shows up in context engineering generally: what you leave out of the prompt matters as much as what you put in. It's also why the retrieval layer deserves the same scrutiny as any other input path, since indirect prompt injection arrives through retrieved documents rather than through your users.

---

A Decision Framework You Can Apply Today

Walk this top to bottom for any new LLM feature.

Step 1: How large is your corpus?

• Under 100K tokens total: skip retrieval, use long context.
• 100K to 1M tokens: depends on freshness, go to Step 2.
• Over 1M tokens: retrieval is required.

Step 2: How fresh does the data need to be?

• Hourly or faster: retrieval. Rebuilding long prompts is too expensive.
• Daily to weekly: either pattern works.
• Static: long context with prompt caching is cheap and clean.

Step 3: What's the query shape?

• Single-document deep reasoning: lean long context.
• Multi-document synthesis: lean hybrid.
• Lookup or fact retrieval: lean classic RAG.
• Exploratory: long context if the doc set is bounded, otherwise hybrid.

Step 4: Do you need citation or access control?

• Yes to either: retrieval is required. Retrofitting citations and per-user filtering onto a long-context-only design is painful.

Step 5: What's your latency budget?

• Under 1 second: classic RAG.
• 1 to 5 seconds: hybrid is feasible.
• Over 5 seconds: any pattern works.

Step 6: What's your accuracy floor on long queries?

• High accuracy on multi-step reasoning past 50K tokens: hybrid with a reranker.
• Best effort: classic RAG is usually fine.

Step 7: Where does your assembled prompt land against the price cliff?

• Under 272K on OpenAI, under 200K on Gemini 3.1 Pro, under 100K on Claude Haiku 5.5, or under your measured safe budget elsewhere: fine.
• Hovering near one of those lines: add a reranker and cut the retrieval set. You'll usually get a better answer and a smaller bill at the same time.

Step 8: Will this run as a long session or an agent?

• Yes: decide up front what gets cleared (old tool results), what gets compacted, what lives outside the conversation (rules, progress), and when the agent starts fresh. Log early quits.
• No: single-shot requests only need Steps 1 to 7.

Most production systems land on hybrid, because real workloads carry at least one constraint that breaks pure long context (multi-tenancy, freshness, cost, citation) and at least one that breaks pure top-k RAG (single-doc reasoning, cross-context queries, exploration).

There's a human version of this same skill. Deciding what deserves to go in front of a reasoning process is what careful readers have always done by hand, and highlighting is that decision made explicit. Glasp's web highlighter keeps the passages you judged worth keeping instead of the whole page, which is retrieval with a human reranker. If you want to point your own tooling at that set, Glasp's MCP connector exposes your highlights to an LLM directly. We wrote more about that setup in turning your notes into an MCP server.

---

Frequently Asked Questions

What is context rot in LLMs?

Context rot is the measured drop in how well an LLM uses its input as that input grows, well before the context window is full. Accuracy on retrieval and reasoning declines unevenly, by different amounts for different models, and faster when distractor text resembles the answer. Chroma found that a coherent haystack hurt more than a shuffled one across all 18 models it tested, and Anthropic's documentation now defines the term directly: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot." Filling a 1M-token window does not buy you a 1M-token-quality answer.

How do you fix context rot?

Send fewer, better tokens. For apps and agents, that means retrieving and reranking instead of stuffing, putting long documents before the question, clearing stale tool results, compacting before the window fills, keeping rules and progress outside the conversation, and handing exploration to sub-agents. For chat and coding tools, it means one chat per task, restarting after two failed corrections, compacting early with a focus instruction, and keeping persistent rules in a file the agent rereads, such as CLAUDE.md.

Do newer models like Claude Opus 5.5 or GPT-6 fix context rot?

No model is immune, but the spread between models is wide. OpenAI reports GPT-6 Astra at 96.3 percent on its 8-needle MRCR between 512K and 1M tokens, while Google's model card shows Gemini 3.1 Pro at 26.3 percent at 1M on Google's own version of the test. The datasets differ, but the gap is too large to ignore. Anthropic's documentation describes context rot as a general property of growing inputs, and Claude Code's docs say performance "degrades as it fills." Measure the model you actually use, on inputs that look like yours.

Is RAG still relevant in 2026?

Yes. Retrieval is still required for corpora that exceed any window, for data that changes hourly, for per-tenant access control, and for citation. What is outdated is classic top-5 RAG as the only pattern. The current default is hybrid: retrieve a bounded set, then reason over all of it. Bigger context windows changed how much you retrieve, not whether you retrieve.

Does long context replace RAG?

Not in the general case. Chroma's Context Rot report showed performance degrading long before the window fills, and vendors build as if they agree: Anthropic's threshold compaction defaults to 150K on 1M-token models, OpenAI reprices requests above 272K input tokens, and Google charges more for Gemini 3.1 Pro prompts above 200K. Long context does replace RAG for one bounded document that fits inside your measured safe budget.

How big should my retrieval set be before context rot kicks in?

Test your specific model, but a reasonable starting point is 20 to 40 percent of the documented window. That's 40K to 80K for a 200K model and 200K to 400K for a 1M model, though on OpenAI you'll want to sit under 272K for billing reasons, on Gemini 3.1 Pro under 200K, and on Claude Haiku 5.5 under 100K. Build a small eval of multi-hop questions, measure accuracy across context sizes, and take the largest size that still clears your bar.

Does prompt caching fix context rot?

No. Caching fixes cost, not accuracy. Cached reads now run between a tenth and a fortieth of the normal input price depending on the model, so long-context queries against stable content get much cheaper and RAG's cost advantage narrows. The model still reads the same long context and degrades the same way, so you're paying less for the same weaker answer. On OpenAI, cached tokens also double in price above 272K.

Should I use a reranker before sending to long context?

For most production hybrid systems, yes. A cross-encoder scoring your top 50 to 200 retrieved chunks sharply improves what reaches the reasoning stage. Skipping rerank usually means stuffing more tokens to make up for weaker precision, which pushes you toward both the rot zone and the price cliff. It's one of the highest-impact changes you can make to a hybrid pipeline.

My agent stops early on long tasks. Is that context rot?

Probably, and it has a name now. The 2026 paper Diagnosing and Mitigating Context Rot in Long-horizon Search calls it premature termination: under heavy context, models give up or hand back a wrong answer they flag as uncertain before exhausting the window, at a rate that climbs with context length. Context management (compaction, trimming, or sub-agents) helps mainly by cutting that rate so the agent keeps exploring. Instrument early quits as their own metric.

---

Closing Thoughts

Every release with a bigger window carries the same implied promise: stop engineering, just dump. Chroma put hard numbers on why that promise hasn't landed, the 2026 work added a second failure mode nobody was watching for, and the underlying math (softmax dilution, position extrapolation, training distribution) says it won't land cleanly at 100M tokens either.

What's left is the boring, productive answer. Build retrieval. Tune it. Add a reranker. Clear what the model no longer needs, and keep what it must never lose outside the conversation. Pick a safe context budget by measuring rather than trusting a spec sheet, and re-measure when the model or its tokenizer changes, which between September and early October 2026 happened at Anthropic, OpenAI, and Google alike. Send the smallest, most relevant set of tokens that contains the answer. Let the model reason over that. Cite the sources.

The 2026 wrinkle is that the vendors now engineer as if they agree with you. OpenAI's line at 272K, Google's at 200K on Gemini 3.1 Pro, Haiku 5.5's at 100K, and a compaction default at 150K on million-token Claude models are all admissions that the usable window is a fraction of the advertised one. Anthropic's flat pricing on its larger models removes the price cliff but not the accuracy cost, and fewer tokens still cost less at a flat rate. So the disciplined architecture is also the cheap one, and those two rarely point the same direction.

If you want a broader map of which model to point at which job, we keep one in the AI task and model matrix. Architecture decisions outlast model releases. Get these right and the next upgrade is a free improvement instead of a forced rewrite.