The Year 2M-Token Windows Stopped Mattering
For about six months in late 2024 and early 2025, a familiar argument made the rounds in engineering Slack channels: RAG is obsolete now. Just paste everything in.
The reasoning sounded clean. Anthropic shipped Claude Sonnet with a 1M-token context window. Google pushed Gemini 2.5 and 2.5 Pro to 2M tokens. Meta announced Llama 4 Scout with a 10M-token theoretical limit. If your knowledge base fits inside one prompt, why bother with vector stores, embedding pipelines, and chunking strategies?
Then in July 2025, Chroma Research published "Context Rot: How Increasing Input Tokens Impacts LLM Performance" by Kelly Hong, Anton Troynikov, and Jeff Huber. The paper ran careful experiments across 18 frontier models and showed something the marketing pages did not: long context windows degrade in non-obvious ways well before they hit their limit. A 200K-token window can show serious accuracy loss at 50K tokens of input. A 1M-token window does not reliably reason across 1M tokens.
That result reframed the architecture conversation. RAG is not obsolete. Long context is not a free lunch. The interesting question is no longer "which one wins," but "which pattern fits this data shape, latency budget, and freshness requirement."
This piece is the architecture-level answer: when should your system retrieve, when should it dump, and when should it do both.
What the Chroma Paper Actually Showed
It's worth reading the Chroma paper carefully because the headline ("context rot exists") understates what's actually there. The team ran extended versions of needle-in-a-haystack (NIAH) benchmarks across GPT-4.1, the Claude 4 family including Opus 4 and Sonnet 4, Gemini 2.5, and Qwen3. Their open replication kit on GitHub lets you reproduce the runs.
Three findings stand out.
Finding 1: Performance degrades non-uniformly as input length grows. This is the core result. Models do not slowly and predictably get worse at a linear rate as input gets longer. They hit cliffs. Some models do fine at 32K and collapse at 64K. Others hold together until they suddenly do not. The size of the documented context window correlates weakly with how well the model actually uses that context.
Finding 2: Semantic similarity drives decay more than length does. When the "needle" is semantically distinct from the surrounding "haystack," models find it fine. When distractors are semantically similar to the answer, accuracy drops sharply, and the drop grows worse with length. In other words: the harder it is to tell the answer apart from the noise, the faster long context falls apart. This matches a separate result from Liu et al. (2024), "Lost in the Middle: How Language Models Use Long Contexts" in TACL, which showed a U-shaped performance curve where the middle of long contexts is systematically underweighted.
Finding 3 (the surprising one): Structured, coherent text degrades attention MORE than shuffled text. This is the result that should change how engineers think. The intuition was that a clean, well-organized 100K-token document is easier for a model to reason over than a jumbled 100K-token blob. The data says the opposite, or at least, not always. Coherent text appears to spread attention more diffusely across a long sequence, while shuffled text creates more distinct local signals that the model can latch onto. The implication: feeding a model a tidy, well-formatted PDF is not automatically safer than feeding it a messy retrieval set.
The Sequential-NIAH benchmark (arXiv 2504.04713) extends this further by testing whether models can chain multiple retrievals from different parts of a long context. The drop-off is even steeper for multi-step reasoning across distance.
Hamel Husain summarized the practical implication well in his notes on context rot: the engineering posture should not be "fit it all in," it should be "give the model the smallest, most relevant context that answers the question."
Why Long Context Fails: The Mechanism
The mechanism matters because it predicts which workloads will fail.
Standard transformer attention uses a softmax over all token pairs. As sequence length N grows, attention weights spread across more positions. Even with relative position encodings like RoPE (rotary position embeddings) or ALiBi, the softmax denominator grows and the per-position weight on any single token shrinks. At 1M tokens, the "right" token has to compete with 999,999 others for a finite attention budget.
Position encodings help, but they're not magic. RoPE has a known degradation curve when extrapolating far beyond training length. Models trained on sequences up to 32K tokens that are deployed at 1M tokens are doing extrapolation that the underlying math does not fully support. Tricks like YaRN, position interpolation, and NTK-aware scaling extend the usable range, but none produce a model that uses 1M tokens as effectively as it uses 32K.
There's also a training data problem. Even when a model trains on long sequences, examples that require cross-context reasoning across 800K tokens are rare. Models learn to use the parts of context that training data taught them to use.
Context rot is not a bug the next release will fix. It's a property of the architecture and training distribution. Future models will push limits further, but the basic pattern will persist for a while.
Where RAG Still Wins
Given context rot, retrieval-augmented generation is not legacy infrastructure. Here's where it keeps winning in 2026.
Multi-document corpora at scale. If your knowledge base is 50,000 documents totaling 500M tokens, you cannot fit it in any context window. Retrieval is the only viable architecture.
Freshness and recency. Vector stores update incrementally. A long-context prompt requires rebuilding the prompt every time content changes. For documents that update hourly (news, catalogs, support tickets, code repos), retrieval handles change cheaply.
Cost. Inference cost scales roughly linearly with input tokens. Sending 200K tokens costs 200x what 1K tokens does. If 95% of your queries can be answered from 5K relevant tokens, retrieval gives you a 40x cost reduction with no accuracy hit.
Citation and provenance. Retrieval gives you a structured list of source documents you can show, link, and rank. Long-context outputs are harder to ground in specific sources without extra citation-extraction plumbing.
Access control and tenancy. If your corpus has per-user, per-tenant, or per-role visibility, you cannot just dump it all in. Retrieval filters by access policy before the model sees the data. Non-negotiable for B2B products.
Multi-corpus reasoning. When the answer combines a Slack message, a Notion page, a Linear issue, and a GitHub PR, retrieval is the bridge.
If your product checks any of these boxes, RAG is not optional. The question becomes how to make retrieval good, not whether to do it.
Where Long Context Wins
Long context has workloads where it's the right answer.
Single-document deep reasoning. Reading a 100-page legal contract and answering questions across clauses. Analyzing a research paper. Summarizing an earnings call. When the right answer connects two paragraphs 80 pages apart, retrieval often breaks the connection. Long context preserves it.
Code understanding within a repository. Many code reasoning tasks need imports, types, definitions, and call sites simultaneously. Chunking by file loses inter-file relationships. Putting the whole repository in context (when it fits) preserves the structure.
Conversational continuity. Long-running agent sessions benefit from full history in context. Retrieval over conversation history is brittle: you often need the last 50 turns, not the most semantically similar 50 turns.
Exploratory reasoning where you don't know the query. If you're not sure what to ask of a document until you've started reasoning about it, retrieval is hard to use. You can't write the query in advance. Long context lets the model browse the material.
Cross-reference within a coherent unit. A textbook chapter, a research paper, a legal brief: these are logically unified. Chunking and reassembling often loses the argument structure.
Rough heuristic: if your data is one logical document and it fits, long context is the cleaner architecture.
The Hybrid Pattern That Actually Works
The 2026 default for serious LLM systems is neither pure RAG nor pure long context. It's a hybrid: retrieve a substantial but bounded set of tokens, then long-context-reason over them.
Here's the canonical flow:
User query
|
v
[Retrieval Stage]
- Vector search (top 100 chunks)
- Optional keyword/BM25 search merged in (hybrid retrieval)
- Optional reranker (cross-encoder over top 100, keep top 30)
|
v
[Assembly Stage]
- Concatenate retrieved chunks
- Add metadata, source headers, structural hints
- Target total: 50K to 200K tokens
|
v
[Long-Context Reasoning Stage]
- Send to frontier model with reasoning prompt
- Model uses the full retrieval set as its context
|
v
Answer + citations
Why this works: each stage handles the failure mode of the other. Retrieval narrows a corpus that's too big for any context window down to a manageable set. Long-context reasoning over the retrieval set restores the multi-document, multi-chunk reasoning that pure RAG often breaks when it sends only the top 5 chunks.
The key engineering decision is the retrieval-set size. Send too few tokens and you lose the long-context benefit (you might as well do classic top-5 RAG). Send too many and you trigger context rot. The Chroma data suggests that the safe ceiling is well below the model's documented limit, often by 4x to 10x. A 200K-window model is usually solid up to 40K to 80K. A 1M-window model can often handle 150K to 300K with minimal rot.
This is the architecture pattern that most teams shipping LLM features at scale converged on by late 2025. It's not glamorous, but it works.
Tuning the Hybrid: Numbers and Heuristics
Let me put numbers to the dials. These are not universal truths; they're starting points that work in production for many teams.
Chunk size. 500 to 1,500 tokens per chunk for most prose. 200 to 500 for code (per function or per logical block). 1,500 to 3,000 for legal or academic text where context within a chunk matters. Overlap chunks by 10 to 20 percent.
Top-k retrieval. Pull more than you'll send. Retrieve top 50 to 200, then rerank. The reranker (a cross-encoder, often a small specialized model like Cohere Rerank or a fine-tuned BGE reranker) costs more per pair than the embedding model but is dramatically more accurate at fine-grained relevance.
Rerank-to-context ratio. After reranking, keep the top 20 to 100 chunks for the long-context stage. The exact number depends on chunk size and your safe-context budget.
Hybrid retrieval. Combine dense (vector) and sparse (BM25, SPLADE) retrieval with reciprocal rank fusion. Pure dense misses exact-match queries (SKUs, error codes, proper nouns). Pure sparse misses paraphrases. Hybrid catches both.
Safe-context budget. Test your model. Build a small eval set of questions whose answer requires reasoning across multiple chunks at different context lengths. Measure accuracy at 16K, 32K, 64K, 128K, 256K tokens of stuffed context. Pick the largest size where accuracy is still acceptable. That's your safe budget. Stay 20 percent under it in production to leave headroom.
Bypass retrieval entirely. "Summarize the document I just uploaded" is a single-document task. Detect these queries with a small classifier and skip retrieval; saves latency and avoids surfacing unrelated noise.
Summarization layer. For very long history (multi-month conversations, large codebases), interpose a summarization step that compresses older material before assembly. Summaries cost tokens too, so test whether they help.
Here's a comparison table that captures the tradeoffs:
| Axis | Pure RAG (top-5 chunks) | Pure Long Context | Hybrid (retrieve 50K-200K, then reason) |
|---|---|---|---|
| Data shape | Many docs, broad corpus | One doc or small set | Many docs, deep reasoning |
| Typical input size | 2K-10K tokens | 100K-2M tokens | 50K-200K tokens |
| Latency | Fast | Slow | Medium |
| Cost per query | Low | High | Medium |
| Accuracy at scale | Good if top-k is right | Degrades with rot | Best for complex queries |
| Freshness | Easy (update index) | Hard (rebuild prompt) | Easy (update index) |
| Citation | Native | Requires extra work | Native (via retrieved set) |
| Access control | Native (filter at retrieval) | Hard | Native |
| Single-doc reasoning | Often breaks | Strong | Strong |
| Cross-doc reasoning | Limited (only top-k) | N/A unless one doc | Strong |
Anti-Patterns Engineers Keep Shipping
A few traps recur often enough to call out.
"Just dump everything in context." Tempting after every release that doubles the window. The Chroma data says it degrades silently. You'll pass spot-checks and fail in production on queries that need cross-context reasoning. Always run an eval at your target context size before shipping.
"Always use RAG." Reflexive RAG for every workload misses single-document cases. Putting a 50-page PDF into a vector index, then retrieving top-5 chunks usually produces a worse answer than feeding the whole PDF to the model. Heuristic: if a single document fits your safe-context budget and the query is about that document, skip retrieval.
"Ignore the retrieval-set token count." Teams set top-k to "however many fits" and discover three months later that their average prompt is 350K tokens and accuracy has quietly degraded. Track assembled context size as a first-class metric. Alert on it.
"Trust the documented window." A 1M-token window does not mean the model uses 1M tokens equally well. The documented limit is what you can technically send. The usable limit is where the model still performs at your quality bar. They're different numbers, often by an order of magnitude.
"Skip evals because the model is good now." Frontier model upgrades change the optimal architecture choice. A workload that needed RAG on GPT-4-32K might do fine on long context with Gemini 2.5 Pro. Re-evaluate when models change.
What 2026 Hardware Changes
Bigger windows do shift some decisions. Being specific about which ones matters.
Llama 4 Scout at 10M tokens. If usable context is even half the documented limit, entire mid-size corpora fit in one prompt. Single-tenant assistants over a 1M-token internal knowledge base can skip retrieval. The economics matter though: 5M tokens of input on a frontier model is expensive per query.
Gemini 2.5 / 3 Pro at 2M tokens. A 2M window with prompt caching shifts the calculus for workflows that repeatedly query the same large document set. Cache 1.5M tokens of background context, pay only for the marginal query, and per-query cost starts to compete with retrieval.
Claude Sonnet 1M. Useful for agentic workloads where session state grows large. Conversational agents that had to summarize or RAG over their own history can now hold more raw history in-context.
Prompt caching across vendors. Anthropic, Google, and OpenAI all support input caching. Long-context architectures get much cheaper for repeated queries against stable content. RAG's cost advantage shrinks when caching kicks in, though it doesn't disappear.
What hasn't changed: context rot. None of these releases include benchmark data showing that bigger windows solve the rot problem. Larger windows raise the ceiling, but the same shape of degradation persists. You still need to measure your safe budget. You still need retrieval for fresh, multi-tenant, multi-source workloads. Hybrid remains the right default.
A Decision Framework You Can Actually Apply
Here's a flow you can apply to any new LLM feature. Walk through it from top to bottom.
Step 1: How large is your corpus?
- Under 100K tokens total: skip retrieval, use long context directly.
- 100K to 1M tokens: depends on freshness (go to Step 2).
- Over 1M tokens: retrieval is required.
Step 2: How fresh does the data need to be?
- Updates per hour or faster: retrieval. Long-context prompts are too expensive to rebuild.
- Updates per day to week: either pattern works.
- Static or rarely updated: long context with prompt caching is cheap and clean.
Step 3: What's the query shape?
- Single-document deep reasoning: lean long context.
- Multi-document synthesis: lean hybrid.
- Lookup or fact retrieval: lean classic RAG (top-k, small context).
- Exploratory or open-ended: lean long context if the doc set is bounded; otherwise hybrid.
Step 4: Do you need citation or access control?
- Yes to either: retrieval is required (either classic RAG or hybrid). Long-context-only architectures are very hard to retrofit with citations and per-user filtering.
Step 5: What's your latency budget?
- Under 1 second: classic RAG (small context, fast).
- 1 to 5 seconds: hybrid is feasible.
- Over 5 seconds: any pattern works.
Step 6: What's your accuracy floor on long queries?
- High accuracy on multi-step reasoning across 50K+ tokens: hybrid with a reranker.
- Best-effort: classic RAG is usually fine.
Walk these six steps and you'll land on one of three architectures: classic top-k RAG, pure long context, or hybrid. Most production systems for serious workloads land on hybrid. Not because hybrid is universally better, but because real workloads usually have at least one constraint (multi-tenant data, freshness, cost, citation) that breaks pure long context, and at least one (single-doc reasoning, cross-context queries, exploration) that breaks pure top-k RAG.
The Chroma paper changed the discourse. It made the long-context-versus-RAG argument feel a little embarrassing in retrospect. They're not competitors; they're two of the three components of a working LLM stack, and the third (reranking) is what makes the hybrid stable.
Frequently Asked Questions
Did Gemini's 2M context window kill RAG?
No. The 2M-token window is real, but the Chroma "Context Rot" paper demonstrated that performance degrades long before the window fills. Practical safe-context budgets for 2M-window models tend to land at 150K to 400K tokens for high-accuracy workloads, which is much less than the marketed limit. RAG (or the hybrid retrieve-then-reason pattern) remains the right architecture for large, multi-document, fresh, or multi-tenant corpora.
What is context rot in plain terms?
Context rot is the observation that LLMs use long context worse than the marketing materials suggest. As you feed in more tokens, accuracy on retrieval and reasoning tasks degrades non-linearly. It gets worse faster when distractor text is semantically similar to the answer, and even coherent, well-structured input can hurt attention more than shuffled input does. The result: filling a 1M-token window does not give you a 1M-token-quality answer.
How big should my retrieval set be before context rot kicks in?
Test your specific model. A rough starting point: stay at 20 to 40 percent of the documented context window for high-accuracy workloads. For a 200K-window model, that's 40K to 80K tokens of retrieval. For a 1M-window model, 200K to 400K. Build a small eval set of multi-hop reasoning questions and measure accuracy across context sizes. Pick the largest size where accuracy is still in your quality range.
Does prompt caching change the calculus?
Yes, meaningfully. Prompt caching (available across Anthropic, Google, and OpenAI) makes the marginal cost of long-context queries against stable content much cheaper. If you can cache a large background document set and only pay for the per-query delta, long context becomes economically competitive with retrieval for static-ish corpora. Caching does not, however, fix context rot: cached long context is still long context, with the same accuracy degradation.
Should I use a reranker before sending to long context?
For most production hybrid systems, yes. A reranker (cross-encoder model that scores query-document pairs) over your top 50 to 200 retrieved chunks dramatically improves the relevance of what you pass to the long-context stage. Skipping rerank often means stuffing more tokens to compensate for lower precision, which pushes you toward the rot zone. Rerank is one of the highest-leverage improvements you can make to a hybrid pipeline.
Closing Thoughts
The seductive thing about every new model release with a bigger window is the implied promise: "stop engineering, just dump." The Chroma paper put hard numbers on why that promise hasn't been delivered, and the underlying math (softmax dilution, position encoding extrapolation, training distribution) suggests it won't be delivered cleanly even when context windows grow to 100M tokens.
What we're left with is the boring, productive answer. Build retrieval. Tune it. Add a reranker. Pick a safe context budget by measuring, not by trusting the spec sheet. Send the model the smallest, most relevant set of tokens that actually contains the answer. Let it reason over that. Cite the sources.
This is less exciting than "the AGI is here, just paste your codebase in," but it ships features that work in production. The teams quietly building good LLM products in 2026 are the ones who treated the long-context narrative with skepticism and the RAG narrative with the same skepticism, and ended up with hybrid pipelines that handle the workloads they actually have.
Architecture decisions outlast model releases. Get those right and the next model upgrade is a free improvement instead of a forced rewrite.