What Context Rot Is in LLMs
Context rot is the measured tendency of a large language model to use information worse as the input gets longer, well before the context window is full. Chroma Research put numbers on it across 18 frontier models in July 2025, and follow-up work through 2026 has sharpened rather than softened the finding. A model advertised at 1M tokens does not reason across 1M tokens at the quality it shows at 30K.
The practical version, for anyone deciding what to build: the documented context window is what you're allowed to send. Your usable window is where the model still hits your quality bar. Those are different numbers, and the second is usually a fraction of the first.
That gap is why the argument that ran through engineering channels in late 2024 and early 2025 ("RAG is obsolete now, just paste everything in") aged badly. The windows did arrive. Anthropic ships 1M tokens on Claude Opus 5 and Claude Sonnet 5. OpenAI's GPT-5.6 family carries about 1.05M. Google's Gemini 3.1 Pro carries 1M. Meta's Llama 4 Scout documents 10M. What didn't arrive was a model that uses all of it well.
So the interesting question stopped being "which one wins" and became "which pattern fits this data shape, this latency budget, this freshness requirement, and this bill." That's what the rest of this piece answers.
What Chroma's Context Rot Research Showed
The headline ("context rot exists") undersells the work. In Context Rot: How Increasing Input Tokens Impacts LLM Performance, published July 14, 2025, Kelly Hong, Anton Troynikov, and Jeff Huber ran controlled experiments across 18 models, including the Claude 4 family plus Claude 3.7 and 3.5, o3, the GPT-4.1 family, GPT-4o, GPT-4 Turbo, Gemini 2.5 Pro and Flash, and three sizes of Qwen3. Their replication kit on GitHub lets you rerun it.
Four findings matter for architecture.
Performance degrades non-uniformly as input grows. If you came here looking for the context rot graph, the shape is the point: it isn't a gentle linear slope. Accuracy holds, then falls off a cliff, and the cliff sits at a different place for every model. Plot it and you get a jagged descent, not a ramp, which is exactly why a spot check at one context size tells you almost nothing about behavior at another.
The size of the drop is large. On LongMemEval, a conversational question-answering benchmark, the team filtered down to 306 prompts and compared two versions of each. The focused version contains only the relevant material and averages about 300 tokens. The full version buries the same answer in the surrounding conversation and averages about 113k tokens. Models handled the focused prompts well and degraded consistently on the full ones. The gap varies a lot by model, and Chroma noted the Claude models showed the most pronounced version of it. Same question, same model, same answer sitting in the input. The only change was how much irrelevant text came along.
Semantic similarity drives decay more than length does. When the "needle" is easy to tell apart from the "haystack," models find it. When distractors look semantically like the answer, accuracy drops sharply, and the drop widens with length. Even a single distractor reduces performance against the needle-only baseline, and four compound it. This lines up with Liu et al. (2024), "Lost in the Middle," in TACL, which found a U-shaped curve where the middle of a long context is systematically underweighted.
Structured, coherent text degrades attention more than shuffled text. This is the result that should change how engineers think. The intuition says a clean 100K-token document is easier to reason over than a jumbled one. Chroma found the opposite, consistently, across all 18 models: shuffling the haystack and destroying its logical flow improved performance. Why that happens is still open, and Chroma only goes as far as saying the structure of an input could influence how attention gets applied. The actionable part needs no mechanism: a tidy, well-formatted PDF is not automatically the safe option.
One more detail to carry into production: failure modes are model-specific. Under distractors, Claude models showed the lowest hallucination rates and tended to abstain when uncertain, while GPT models showed the highest and tended to answer confidently and wrongly. If you're choosing between them, you're partly choosing which failure you'd rather debug.
| What you change | What happens to accuracy | What it means for your pipeline |
|---|---|---|
| Input grows from ~300 to ~113k tokens | Consistent degradation on LongMemEval | Irrelevant context is not free padding |
| Needle looks like the haystack | Sharp drop, worsens with length | Rerank for precision, don't just add tokens |
| One distractor added | Measurable drop from baseline | Cut near-misses before assembly |
| Haystack shuffled vs coherent | Shuffled scored better across all 18 models | Clean formatting is not a safety margin |
| Model swapped under distractors | Claude abstains, GPT hallucinates | Choose the failure mode you can detect |
The Sequential-NIAH benchmark (arXiv 2504.04713) pushes on the same seam from another angle, testing whether models can extract needles that have to come back in the right order. Across six LLMs on contexts from 8K to 128K, the best model managed 63.5 percent. Multi-step retrieval across distance is harder than the single-needle demos suggest.
For a gentler entry point, Hamel Husain hosted Kelly Hong to walk through the research and published an annotated write-up of that talk. Its takeaways land where the report does: performance isn't uniform across context lengths, how you present information matters, and deliberate context engineering is the lever.
Why Long Context Fails: The Mechanism
The mechanism matters because it predicts which workloads break.
Transformer attention runs a softmax over token pairs. As sequence length grows, attention spreads across more positions. Even with relative position encodings like RoPE or ALiBi, the softmax denominator grows and the weight available for any single token shrinks. At 1M tokens, the right token competes with 999,999 others for a finite budget.
Position encodings stretch the range without removing the problem. RoPE degrades when extrapolating far past training length, so a model trained on 32K sequences and deployed at 1M is doing extrapolation the underlying math doesn't fully support. YaRN, position interpolation, and NTK-aware scaling all help, and none of them produce a model that uses 1M tokens as well as it uses 32K.
There's a training data problem too. Even when a model trains on long sequences, examples requiring genuine reasoning across 800K tokens are rare. Models learn to use the parts of context their training data taught them to use.
So context rot is a property of the architecture and the training distribution rather than a bug the next release patches out. Future models will push the frontier further, and the shape of the degradation will persist.
Premature Termination: What 2026 Research Added
The most useful new result since this article first published came out of long-horizon agent work rather than benchmark work.
In Diagnosing and Mitigating Context Rot in Long-horizon Search (June 2026), Shijie Xia, Yikun Wang, Zhen Huang, and Pengfei Liu studied four flagship models across three benchmarks and named a failure mode the earlier literature had skipped: premature termination. Under a large context, models give up, or hand back an uncertain wrong answer, long before they've exhausted the window. Controlling for how hard the query was, they found the premature termination rate rises with context length.
That reframes the problem. Context rot covers two different failures, and they need opposite fixes. If the model can't find the needle anymore, improve precision. If the model stops looking, it needs room and reason to keep exploring. For an agent doing multi-step search, conflating them wastes weeks.
Their second finding is the one to steal. They analyzed seven context-management methods across three categories and concluded these methods work mainly as test-time scaling strategies: by cutting the premature termination rate, they buy the model more exploration. Compaction and pruning earn their keep as accuracy features, not just cost controls. The authors also built a behavior-aware filtering strategy for parallel sampling and measured a 2.6 to 4.9 percent gain across three aggregation methods.
If you run agents over long sessions, track how often yours quit early. It's a cheap metric and almost nobody instruments it.
Where RAG Still Wins
Given all that, retrieval-augmented generation still earns its place. Here's where it keeps winning.
Multi-document corpora at scale. If your knowledge base is 50,000 documents totaling 500M tokens, it fits in no window. Retrieval is the only viable architecture.
Freshness and recency. Vector stores update incrementally. A long-context prompt has to be rebuilt whenever content changes. For anything updating hourly (news, catalogs, support tickets, code), retrieval handles change cheaply.
Cost. Input cost scales with input tokens, and above certain thresholds it scales worse than linearly. If 95 percent of your queries can be answered from 5K relevant tokens, retrieval is a large multiple cheaper with no accuracy loss.
Citation and provenance. Retrieval hands you a structured list of sources you can show, link, and rank, while grounding a long-context answer in specific sources takes extra plumbing. This is the same reason a reading tool that keeps the passage and its source beats one that keeps a summary: when you ask questions across everything you've saved in Glasp, every answer can point back at the highlight it came from.
Access control and tenancy. If your corpus has per-user, per-tenant, or per-role visibility, you can't dump it all in. Retrieval filters by policy before the model sees anything. That's non-negotiable for B2B.
Multi-corpus reasoning. When the answer spans a Slack thread, a Notion page, a Linear issue, and a GitHub PR, retrieval is the bridge.
Check any of those boxes and RAG isn't optional. The question becomes how to make retrieval good, not whether to do it.
Where Long Context Wins
Long context has workloads where it's simply the right answer.
Single-document deep reasoning. Reading a 100-page contract and answering across clauses. Analyzing a paper. Working through an earnings call. When the answer connects two paragraphs 80 pages apart, chunking often severs the link.
Code understanding inside a repository. Plenty of code tasks need imports, types, definitions, and call sites at once. Chunking by file loses the relationships between files.
Conversational continuity. Long agent sessions benefit from real history. Retrieval over conversation history is brittle, because you usually need the last 50 turns, not the 50 most semantically similar ones.
Exploratory reasoning where you don't know the query yet. If you can't write the query in advance, retrieval is hard to aim. Long context lets the model browse.
Cross-reference within a coherent unit. A textbook chapter, a paper, a legal brief. Chunking and reassembling these tends to lose the argument.
Rough heuristic: if your data is one logical document and it fits inside your measured safe budget, long context is the cleaner architecture.
The Hybrid Pattern Most Teams Land On
The 2026 default for serious systems is neither pure RAG nor pure long context. Retrieve a substantial but bounded set of tokens, then reason over them.
User query
|
v
[Retrieval Stage]
- Vector search (top 100 chunks)
- Optional keyword/BM25 search merged in (hybrid retrieval)
- Optional reranker (cross-encoder over top 100, keep top 30)
|
v
[Assembly Stage]
- Concatenate retrieved chunks
- Add metadata, source headers, structural hints
- Target total: 50K to 200K tokens
|
v
[Long-Context Reasoning Stage]
- Send to frontier model with reasoning prompt
- Model uses the full retrieval set as its context
|
v
Answer + citations
Each stage covers the other's failure mode. Retrieval narrows a corpus that's too big for any window down to something manageable. Reasoning over the whole retrieved set restores the cross-chunk reasoning that classic top-5 RAG throws away.
The load-bearing decision is retrieval-set size. Too few tokens and you've just rebuilt top-5 RAG. Too many and you're in the rot zone, paying more for a worse answer. A workable rule of thumb, which the next section turns into a measurement: plan for a safe budget around 20 to 40 percent of the documented window, then verify it on your own data. For a 200K-window model that's 40K to 80K. For a 1M-window model it's 200K to 400K, which, as it happens, is right about where the billing changes.
Tuning the Hybrid: Numbers and Heuristics
These aren't universal truths. They're starting points that hold up in production.
Chunk size. 500 to 1,500 tokens for prose. 200 to 500 for code, per function or logical block. 1,500 to 3,000 for legal or academic text where within-chunk context carries meaning. Overlap by 10 to 20 percent.
Top-k retrieval. Pull more than you'll send. Retrieve top 50 to 200, then rerank. A cross-encoder costs more per pair than an embedding model and is dramatically better at fine-grained relevance.
Rerank-to-context ratio. Keep the top 20 to 100 chunks after reranking. The exact number follows from chunk size and your safe budget.
Hybrid retrieval. Combine dense and sparse (BM25, SPLADE) with reciprocal rank fusion. Dense alone misses exact matches like SKUs, error codes, and proper nouns. Sparse alone misses paraphrases.
Two of these dials deserve more than a bullet, because they're where teams lose accuracy without noticing.
The first is your safe-context budget, and the only honest way to set it is to measure. Build a small eval set of questions that need reasoning across several chunks, then score accuracy at 16K, 32K, 64K, 128K, and 256K of stuffed context. Take the largest size that still clears your bar and run 20 percent under it for headroom. That measured number should land somewhere near the 20 to 40 percent rule above, and when it doesn't, trust your eval over the heuristic.
The second is what happens when you upgrade the model, which bit teams in 2026. Anthropic's tokenizer changed with Claude Opus 4.7, and the same text now maps to roughly 1.0 to 1.35 times as many tokens depending on the content, which Anthropic summarizes as about 30 percent more. Your documents didn't grow. Your budget shrank. If you pinned a token ceiling a year ago and swapped the model underneath it, that ceiling means something different now, and nothing in your logs will announce it.
Bypass retrieval entirely when the query says so. "Summarize the document I just uploaded" is a single-document task. Detect those with a small classifier, skip retrieval, save latency, and avoid surfacing unrelated noise.
Summarization and pruning layers. For very long histories, compress older material before assembly. Summaries cost tokens too, so measure whether they actually help.
| Axis | Pure RAG (top-5 chunks) | Pure Long Context | Hybrid (retrieve 50K-200K, then reason) |
|---|---|---|---|
| Data shape | Many docs, broad corpus | One doc or small set | Many docs, deep reasoning |
| Typical input size | 2K-10K tokens | 100K-1M tokens | 50K-200K tokens |
| Latency | Fast | Slow | Medium |
| Cost per query | Low | High, and worse past the price cliff | Medium |
| Accuracy at scale | Good if top-k is right | Degrades with rot | Best for complex queries |
| Freshness | Easy (update index) | Hard (rebuild prompt) | Easy (update index) |
| Citation | Native | Requires extra work | Native (via retrieved set) |
| Access control | Native (filter at retrieval) | Hard | Native |
| Single-doc reasoning | Often breaks | Strong | Strong |
| Cross-doc reasoning | Limited (only top-k) | N/A unless one doc | Strong |
Anti-Patterns Engineers Keep Shipping
A few traps recur often enough to name.
"Just dump everything in context." Tempting after every release that doubles the window. It degrades silently, so you pass spot checks and fail in production on exactly the queries that needed cross-context reasoning. Run an eval at your target size before shipping.
"Always use RAG." Reflexive retrieval misses single-document cases. Indexing a 50-page PDF and pulling top-5 chunks usually beats nothing and loses to just sending the PDF.
"Ignore the assembled token count." Teams set top-k to "whatever fits," then discover three months later that their average prompt is 350K tokens, accuracy has quietly slipped, and the bill doubled at a threshold nobody read. Track assembled context size as a first-class metric and alert on it.
"Trust the documented window." The documented limit is what you may send. The usable limit is where quality holds. Those are different numbers, and only one of them is on the spec sheet.
"Skip evals because the model is good now." Model upgrades change the right architecture, and sometimes change your token accounting underneath you. Re-evaluate when models change.
"Assume a quiet failure is a safe failure." Under distractors, some models hallucinate and some abstain, and in long-horizon work some just stop early. Know which one yours does, and instrument for it.
What Changed in 2026: Windows, Prices, and Compaction
Two things shifted since the first version of this article, and both cut the same way.
Long context now carries a published price cliff, at two vendors. OpenAI bills the GPT-5.6 family on two meters. Below the threshold, GPT-5.6 Sol runs $4 per million input tokens and $20 per million output. Above it, the same model runs $8 and $30. Terra goes from $2/$12 to $4/$18, and Luna from $0.20/$1.20 to $0.40/$1.80. The threshold sits at 272K input tokens, and crossing it reprices the entire request, not just the overflow. A 272,001-token prompt bills every one of those tokens at the higher rate. Against an advertised 1.05M-token window, that's roughly a quarter of it at the advertised price.
Google does the same thing at a lower threshold. Gemini 3.1 Pro runs $2 per million input and $12 per million output for prompts up to 200K tokens, then $4 and $18 above it. Same 2x input and 1.5x output structure, 72K tokens earlier. If you sized your pipeline against OpenAI's cliff and then switched vendors, you can sail past Google's without changing a line of code.
Two independent vendors pricing long context as a premium is stronger evidence than either one alone. It turns a quality argument into a budget argument: staying under your safe-context budget was already the accuracy-preserving move, and now it's also the difference between one bill and two.
Vendors ship context management as a product feature. Anthropic's compaction summarizes earlier context server-side as a conversation grows. It's opt-in rather than on by default, but once you enable it the trigger threshold defaults to 150K tokens on models whose window is 1M. Read that again: the vendor with a million-token window sets its own default at 15 percent of it. Anthropic also ships context editing, which clears old tool results or thinking blocks rather than summarizing them. Two different mechanisms, same underlying admission.
Prompt caching is the genuine cost lever, and it's real: cached input reads run about a tenth of the normal input price on both major APIs. Caching makes repeated queries against stable content much cheaper. It does not fix context rot, and on OpenAI it doesn't exempt you from the long-context tier either.
| Model | Documented window | Practical caveat |
|---|---|---|
| Claude Opus 5 / Sonnet 5 | 1M tokens | Compaction defaults to 150K once enabled |
| Claude Haiku 4.5 | 200K tokens | Safe budget lands near 40K-80K |
| GPT-5.6 (Sol / Terra / Luna) | ~1.05M tokens | Whole request reprices above 272K input |
| Gemini 3.1 Pro | 1M tokens | Whole request reprices above 200K |
| Llama 4 Scout | 10M tokens (iRoPE) | Documented, not demonstrated at that depth |
What hasn't changed is the underlying result. None of these releases shipped evidence that a bigger window fixes rot. Larger windows raise the ceiling, the shape of the degradation persists, and you still need retrieval for fresh, multi-tenant, multi-source workloads.
This is the same discipline that shows up in context engineering generally: what you leave out of the prompt matters as much as what you put in. It's also why the retrieval layer deserves the same scrutiny as any other input path, since indirect prompt injection arrives through retrieved documents rather than through your users.
A Decision Framework You Can Apply Today
Walk this top to bottom for any new LLM feature.
Step 1: How large is your corpus?
- Under 100K tokens total: skip retrieval, use long context.
- 100K to 1M tokens: depends on freshness, go to Step 2.
- Over 1M tokens: retrieval is required.
Step 2: How fresh does the data need to be?
- Hourly or faster: retrieval. Rebuilding long prompts is too expensive.
- Daily to weekly: either pattern works.
- Static: long context with prompt caching is cheap and clean.
Step 3: What's the query shape?
- Single-document deep reasoning: lean long context.
- Multi-document synthesis: lean hybrid.
- Lookup or fact retrieval: lean classic RAG.
- Exploratory: long context if the doc set is bounded, otherwise hybrid.
Step 4: Do you need citation or access control?
- Yes to either: retrieval is required. Retrofitting citations and per-user filtering onto a long-context-only design is painful.
Step 5: What's your latency budget?
- Under 1 second: classic RAG.
- 1 to 5 seconds: hybrid is feasible.
- Over 5 seconds: any pattern works.
Step 6: What's your accuracy floor on long queries?
- High accuracy on multi-step reasoning past 50K tokens: hybrid with a reranker.
- Best effort: classic RAG is usually fine.
Step 7: Where does your assembled prompt land against the price cliff?
- Under 272K on OpenAI, under 200K on Google, or under your measured safe budget elsewhere: fine.
- Hovering near either threshold: add a reranker and cut the retrieval set. You'll usually get a better answer and a smaller bill at the same time.
Most production systems land on hybrid, because real workloads carry at least one constraint that breaks pure long context (multi-tenancy, freshness, cost, citation) and at least one that breaks pure top-k RAG (single-doc reasoning, cross-context queries, exploration).
There's a human version of this same skill. Deciding what deserves to go in front of a reasoning process is what careful readers have always done by hand, and highlighting is that decision made explicit. Glasp's web highlighter keeps the passages you judged worth keeping instead of the whole page, which is retrieval with a human reranker. If you want to point your own tooling at that set, Glasp's MCP connector exposes your highlights to an LLM directly. We wrote more about that setup in turning your notes into an MCP server.
Frequently Asked Questions
What is context rot in LLMs?
Context rot is the observation that LLMs use long context worse than the marketing suggests. As you feed in more tokens, accuracy on retrieval and reasoning degrades non-linearly, hitting cliffs rather than sliding down a ramp. It gets worse faster when distractor text resembles the answer, and Chroma found that even coherent, well-structured input hurt attention more than shuffled input across all 18 models tested. Filling a 1M-token window does not buy you a 1M-token-quality answer.
Is RAG outdated in 2026?
No, and the evidence points the other way. Retrieval is still required for corpora that exceed any window, for data that changes hourly, for per-tenant access control, and for citation. What is outdated is classic top-5 RAG as the only pattern. The current default is hybrid: retrieve a bounded set, then reason over all of it. Bigger context windows changed how much you retrieve, not whether you retrieve.
Does long context replace RAG?
Not in the general case. Chroma's Context Rot report showed performance degrading long before the window fills, and vendors have since agreed in their own products: Anthropic's opt-in compaction defaults to summarizing at 150K on a 1M-token model, OpenAI reprices requests above 272K input tokens, and Google above 200K. Long context does replace RAG for one bounded document that fits inside your measured safe budget.
How big should my retrieval set be before context rot kicks in?
Test your specific model, but a reasonable starting point is 20 to 40 percent of the documented window. That's 40K to 80K for a 200K model and 200K to 400K for a 1M model, though on OpenAI you'll want to sit under 272K for billing reasons and on Google under 200K. Build a small eval of multi-hop questions, measure accuracy across context sizes, and take the largest size that still clears your bar.
Does prompt caching fix context rot?
No. Caching fixes cost, not accuracy. Cached input reads run roughly a tenth of the normal input price, so long-context queries against stable content get much cheaper and RAG's cost advantage narrows. The model still reads the same long context and degrades the same way, so you're paying less for the same weaker answer. On OpenAI, caching also doesn't exempt you from the long-context price tier.
Should I use a reranker before sending to long context?
For most production hybrid systems, yes. A cross-encoder scoring your top 50 to 200 retrieved chunks sharply improves what reaches the reasoning stage. Skipping rerank usually means stuffing more tokens to make up for weaker precision, which pushes you toward both the rot zone and the price cliff. It's one of the highest-impact changes you can make to a hybrid pipeline.
My agent stops early on long tasks. Is that context rot?
Probably, and it has a name now. The 2026 paper Diagnosing and Mitigating Context Rot in Long-horizon Search calls it premature termination: under heavy context, models give up or answer with unearned confidence before exhausting the window, at a rate that climbs with context length. Context management helps mainly by cutting that rate so the agent keeps exploring. Instrument early quits as their own metric.
Closing Thoughts
Every release with a bigger window carries the same implied promise: stop engineering, just dump. Chroma put hard numbers on why that promise hasn't landed, the 2026 work added a second failure mode nobody was watching for, and the underlying math (softmax dilution, position extrapolation, training distribution) says it won't land cleanly at 100M tokens either.
What's left is the boring, productive answer. Build retrieval. Tune it. Add a reranker. Pick a safe context budget by measuring rather than trusting a spec sheet, and re-measure when the model or its tokenizer changes. Send the smallest, most relevant set of tokens that contains the answer. Let the model reason over that. Cite the sources.
The 2026 wrinkle is that the vendors now price and engineer as if they agree with you. Cliffs at 272K and 200K, and a compaction default at 150K, are all admissions that the usable window is a fraction of the advertised one. That's useful, because it means the disciplined architecture is also the cheap one, and those two rarely point the same direction.
If you want a broader map of which model to point at which job, we keep one in the AI task and model matrix. Architecture decisions outlast model releases. Get these right and the next upgrade is a free improvement instead of a forced rewrite.
