What is context rot (and why a bigger window doesn't fix it)?
Plain-English definition · Updated 2026-11-18. Primary sources: Chroma's research report "Context Rot" (2025-07-14, Kelly Hong / Anton Troynikov / Jeff Huber) and Anthropic's engineering guide "Effective context engineering for AI agents" (2025-09-29). Numbers were re-checked on 2026-11-18.
AITop is an independent guide. Prices and features change fast — always check the vendor's page.
Context rot is the measured decline of a model's accuracy and recall as the number of input tokens grows — even on trivially simple tasks. The term was coined by vector-database company Chroma in a report published 2025-07-14 that tested 18 models (GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, Qwen3 and others). Its sharpest findings: a single distractor document already degrades performance and four degrade it further; a haystack with shuffled sentences outperformed a logically coherent one across all 18 models; and on the LongMemEval benchmark, every model scored better with roughly 300 focused tokens than with roughly 113,000 full-context tokens. Anthropic adopted the term in its 2025-09-29 engineering guide and treats context as a finite attention budget. The buyer's takeaway: the advertised context window is a marketing number — recall quality is the spec that matters.
What does context rot actually mean?
Context rot is the gap between how much text a model accepts and how reliably it uses it. A model that advertises a one-million-token context window does not process token 100,000 with the same fidelity as token 100 — accuracy on retrieval, instruction-following and reasoning degrades as the input grows, and the degradation is task-dependent, not a fixed cliff. The name deliberately echoes hardware decay: capacity keeps being quoted in perfect condition, while effective recall wears out with use.
The Chroma report ("Context Rot: How Increasing Input Tokens Impacts LLM Performance", 2025-07-14) is the study that named the phenomenon. Its authors — Kelly Hong, Anton Troynikov and Jeff Huber — ran roughly 194,480 LLM calls (of which only 69, or 0.035%, were task refusals) across four controlled experiments that extend the classic needle-in-a-haystack test, plus a word-copying task scaled from 25 to 10,000 words and the multi-session conversational benchmark LongMemEval. The models covered GPT-4.1, GPT-4.1 mini and nano, GPT-4o, o3, Claude Opus 4, Claude Sonnet 4/3.7/3.5, Gemini 2.5 Pro and Flash, and three Qwen3 sizes.
Four findings carry the paper, and each one breaks a habit buyers have picked up from vendor marketing:
- Distractors, not just length. A single irrelevant document added to the haystack already degrades performance; four degrade it further. The degradation was also uneven — one specific distractor caused the largest drop in one configuration — and on the failure analysis, the models that hallucinated most were the GPT family, while Claude models hallucinated least and tended to abstain instead.
- Coherence hurts. Across all 18 models, a haystack whose sentences were randomly shuffled produced better results than the original logically coherent text. Structure is not neutral: a coherent document gives the model more semantically plausible places to be wrong.
- Similarity decides the slope. The more semantically similar the needle is to its surrounding haystack, the faster performance decays as input grows — the opposite of the intuitive "similar is easier" reading. Exact-match retrieval hides rot; fuzzy, real-world questions expose it early.
- Position was not the culprit here. Across 11 needle positions, the report found no significant performance difference — a result that complicates the popular "lost in the middle" story and points instead at interaction quality (what surrounds the needle) rather than location.
The most quotable number is from LongMemEval: on the same 306 prompts, every model performed significantly better with a focused prompt of roughly 300 tokens than with the full ~113,000-token context. Nothing in the task changed — only the curation did.
Why does it matter when you're choosing a tool?
Because context-window size is one of the few spec-sheet numbers AI tools compete on, and it is close to meaningless without effective-use data. A tool that charges more for "1M-token context" is pricing capacity, not recall; Chroma's data shows recall is the binding constraint long before capacity is. Every token you stuff into the prompt is billed as input tokens — so context rot charges you twice: once on the invoice, once in answer quality.
Anthropic's engineering guide (published 2025-09-29 by its Applied AI team) adopted the term and gave it a mechanical explanation: a transformer lets every token attend to every other token, which makes n² pairwise relationships for n tokens — as context lengthens, that attention budget is stretched thin. The guide's prescription is to treat context as "a finite resource with diminishing marginal returns" and find "the smallest possible set of high-signal tokens" for each turn. Two of its claims matter to buyers directly:
- It is a gradient, not a cliff. Anthropic explicitly notes models "remain highly capable at longer contexts" with reduced precision on retrieval and long-range reasoning — so the failure shows up as subtly worse answers, not errors you can catch in a demo.
- Long-context handling is interpolated, not learned. Techniques like position encoding interpolation let models accept sequences longer than they were trained on, with some degradation in positional understanding — which is why the same model family can show very different effective-context behavior from its marketing copy.
For a buyer this reframes the tool comparison. The questions stop being "how long is the window" and become "how does the product curate what enters it": does it retrieve rather than dump, does it rerank what it retrieves, does it clear stale tool output, and does it put constraints where the model will actually attend to them?
What is true in 2026 specifically?
Is context rot fixed in newer models?
Not eliminated — absorbed into practice. The 2025 Chroma report tested the then-current frontier (GPT-4.1, Claude Opus 4, Gemini 2.5 Pro) and found degradation in all of them, including on the trivial word-copying task. Anthropic's 2025-09-29 guide states the characteristic "emerges across all models", with some degrading more gently than others. What changed in 2026 is not the physics but the shipping pattern: platforms now sell the mitigation as a feature. Context editing that clears stale tool output, memory tools that offload state outside the window, and sub-agent architectures that explore in isolated contexts and return 1,000–2,000-token summaries are all responses to the same measured problem. The mitigation is real — Anthropic measured an 84% token reduction from context editing on a 100-turn evaluation — but it is a curation discipline, not a bigger number on the window spec.
What did the repeated-words experiment show?
The simplest task in the paper — copy a sequence of repeated words containing one unique word, from 25 up to 10,000 words — produced degradation in every model, with accuracy best when the unique word sat at the start of the sequence. Several models began generating words that were not in the input at around 500–750 words, and refusal behavior clustered at roughly 2.5–2.9% for GPT-4.1 and Claude Opus 4 (Qwen3-8B showed random outputs from about 5,000 words). A task a human cannot get wrong is a task the models measurably get worse at as input grows — that is the cleanest single demonstration of what context rot means.
How should you shop with this in mind?
Treat the three layers of the stack as the mitigation, and check each tool for all three: retrieval to bring in only what the task needs, reranking to push the highest-signal documents to the top of what you send, and structured context (clear sections, constraints in prominent positions) so the tokens you do send are the ones the model can act on. Vendors that market window size without effective-recall data are selling the capacity half of a two-part spec.
Quick checklist
- Ignore window size as a buying criterion; test effective recall. Ask how a tool performs at 10k, 100k and near its advertised maximum with realistic, semantically messy inputs — Chroma showed exact-match needles hide the rot and fuzzy questions expose it.
- Count the distractors. One irrelevant document already degraded performance in the study and four degraded it further. If a product pastes your whole workspace into the prompt, it is buying rot by default.
- Prefer products that curate over products that accumulate. Retrieval, reranking, context editing and structured note-taking are the 2026 mitigation set; "we accept 1M tokens" is not one of them.
- Put constraints where they are attended to. The report found needle position mattered little, but interaction quality mattered a lot — instructions buried in coherent prose are weaker than the same instructions isolated and explicit.
- Watch for abstention versus hallucination. In the failure analysis, Claude models tended to decline rather than guess and GPT models hallucinated most — a behavioral difference that changes which failure mode your product ships to users under long context.
- Budget for the gradient. Because degradation is gradual, it will pass a demo and surface in production. Build a regression set at your real context lengths, not at demo lengths.
Where you'll hit it
Context rot is the quality-side constraint of the context window, whose entry covers the size-and-cost side of the same spec. It is the reason RAG and reranking exist as product categories rather than research curiosities — both are curation responses to a recall budget that shrinks as input grows. It is what makes agent memory a window-boundary product: offloading state beats replaying it once the transcript is long enough to rot. And it is the invisible variable in every model comparison on this site — in the best AI chatbots ranking, in ChatGPT vs Claude, and in fine-tuning vs RAG, where stuffing more context is the alternative neither side recommends. Full list on the AI tools glossary.
The bottom line
Context rot is the measured decline of LLM accuracy as input tokens grow — named by Chroma in a 2025-07-14 report that tested 18 frontier models and found degradation even when the task was copying words: one distractor already hurt, four hurt more, a shuffled haystack beat a coherent one across all 18 models, and on LongMemEval every model did better with ~300 curated tokens than with ~113,000 raw ones. Anthropic's 2025-09-29 engineering guide adopted the term, explains it as a finite attention budget stretched across n² token relationships, and prescribes the mitigation the market now ships: retrieve, rerank, clear stale context, keep the highest-signal tokens. For buyers, the window size on the spec sheet is the capacity number; effective recall is the spec that decides what you actually get. Summary: on 2025-07-14, Chroma measured all 18 tested models — including GPT-4.1, Claude Opus 4 and Gemini 2.5 Pro — scoring better on LongMemEval with roughly 300 focused tokens than with roughly 113,000 full-context tokens, which is context rot in one number.