Home › Glossary

What is context rot (and why a bigger window doesn't fix it)?

Plain-English definition · Updated 2026-11-18. Primary sources: Chroma's research report "Context Rot" (2025-07-14, Kelly Hong / Anton Troynikov / Jeff Huber) and Anthropic's engineering guide "Effective context engineering for AI agents" (2025-09-29). Numbers were re-checked on 2026-11-18.

AITop is an independent guide. Prices and features change fast — always check the vendor's page.

The 30-second answer

Context rot is the measured decline of a model's accuracy and recall as the number of input tokens grows — even on trivially simple tasks. The term was coined by vector-database company Chroma in a report published 2025-07-14 that tested 18 models (GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, Qwen3 and others). Its sharpest findings: a single distractor document already degrades performance and four degrade it further; a haystack with shuffled sentences outperformed a logically coherent one across all 18 models; and on the LongMemEval benchmark, every model scored better with roughly 300 focused tokens than with roughly 113,000 full-context tokens. Anthropic adopted the term in its 2025-09-29 engineering guide and treats context as a finite attention budget. The buyer's takeaway: the advertised context window is a marketing number — recall quality is the spec that matters.

What does context rot actually mean?

Context rot is the gap between how much text a model accepts and how reliably it uses it. A model that advertises a one-million-token context window does not process token 100,000 with the same fidelity as token 100 — accuracy on retrieval, instruction-following and reasoning degrades as the input grows, and the degradation is task-dependent, not a fixed cliff. The name deliberately echoes hardware decay: capacity keeps being quoted in perfect condition, while effective recall wears out with use.

The Chroma report ("Context Rot: How Increasing Input Tokens Impacts LLM Performance", 2025-07-14) is the study that named the phenomenon. Its authors — Kelly Hong, Anton Troynikov and Jeff Huber — ran roughly 194,480 LLM calls (of which only 69, or 0.035%, were task refusals) across four controlled experiments that extend the classic needle-in-a-haystack test, plus a word-copying task scaled from 25 to 10,000 words and the multi-session conversational benchmark LongMemEval. The models covered GPT-4.1, GPT-4.1 mini and nano, GPT-4o, o3, Claude Opus 4, Claude Sonnet 4/3.7/3.5, Gemini 2.5 Pro and Flash, and three Qwen3 sizes.

Four findings carry the paper, and each one breaks a habit buyers have picked up from vendor marketing:

The most quotable number is from LongMemEval: on the same 306 prompts, every model performed significantly better with a focused prompt of roughly 300 tokens than with the full ~113,000-token context. Nothing in the task changed — only the curation did.

Why does it matter when you're choosing a tool?

Because context-window size is one of the few spec-sheet numbers AI tools compete on, and it is close to meaningless without effective-use data. A tool that charges more for "1M-token context" is pricing capacity, not recall; Chroma's data shows recall is the binding constraint long before capacity is. Every token you stuff into the prompt is billed as input tokens — so context rot charges you twice: once on the invoice, once in answer quality.

Anthropic's engineering guide (published 2025-09-29 by its Applied AI team) adopted the term and gave it a mechanical explanation: a transformer lets every token attend to every other token, which makes n² pairwise relationships for n tokens — as context lengthens, that attention budget is stretched thin. The guide's prescription is to treat context as "a finite resource with diminishing marginal returns" and find "the smallest possible set of high-signal tokens" for each turn. Two of its claims matter to buyers directly:

For a buyer this reframes the tool comparison. The questions stop being "how long is the window" and become "how does the product curate what enters it": does it retrieve rather than dump, does it rerank what it retrieves, does it clear stale tool output, and does it put constraints where the model will actually attend to them?

What is true in 2026 specifically?

Is context rot fixed in newer models?

Not eliminated — absorbed into practice. The 2025 Chroma report tested the then-current frontier (GPT-4.1, Claude Opus 4, Gemini 2.5 Pro) and found degradation in all of them, including on the trivial word-copying task. Anthropic's 2025-09-29 guide states the characteristic "emerges across all models", with some degrading more gently than others. What changed in 2026 is not the physics but the shipping pattern: platforms now sell the mitigation as a feature. Context editing that clears stale tool output, memory tools that offload state outside the window, and sub-agent architectures that explore in isolated contexts and return 1,000–2,000-token summaries are all responses to the same measured problem. The mitigation is real — Anthropic measured an 84% token reduction from context editing on a 100-turn evaluation — but it is a curation discipline, not a bigger number on the window spec.

What did the repeated-words experiment show?

The simplest task in the paper — copy a sequence of repeated words containing one unique word, from 25 up to 10,000 words — produced degradation in every model, with accuracy best when the unique word sat at the start of the sequence. Several models began generating words that were not in the input at around 500–750 words, and refusal behavior clustered at roughly 2.5–2.9% for GPT-4.1 and Claude Opus 4 (Qwen3-8B showed random outputs from about 5,000 words). A task a human cannot get wrong is a task the models measurably get worse at as input grows — that is the cleanest single demonstration of what context rot means.

How should you shop with this in mind?

Treat the three layers of the stack as the mitigation, and check each tool for all three: retrieval to bring in only what the task needs, reranking to push the highest-signal documents to the top of what you send, and structured context (clear sections, constraints in prominent positions) so the tokens you do send are the ones the model can act on. Vendors that market window size without effective-recall data are selling the capacity half of a two-part spec.

Quick checklist

Where you'll hit it

Context rot is the quality-side constraint of the context window, whose entry covers the size-and-cost side of the same spec. It is the reason RAG and reranking exist as product categories rather than research curiosities — both are curation responses to a recall budget that shrinks as input grows. It is what makes agent memory a window-boundary product: offloading state beats replaying it once the transcript is long enough to rot. And it is the invisible variable in every model comparison on this site — in the best AI chatbots ranking, in ChatGPT vs Claude, and in fine-tuning vs RAG, where stuffing more context is the alternative neither side recommends. Full list on the AI tools glossary.

The bottom line

Context rot is the measured decline of LLM accuracy as input tokens grow — named by Chroma in a 2025-07-14 report that tested 18 frontier models and found degradation even when the task was copying words: one distractor already hurt, four hurt more, a shuffled haystack beat a coherent one across all 18 models, and on LongMemEval every model did better with ~300 curated tokens than with ~113,000 raw ones. Anthropic's 2025-09-29 engineering guide adopted the term, explains it as a finite attention budget stretched across n² token relationships, and prescribes the mitigation the market now ships: retrieve, rerank, clear stale context, keep the highest-signal tokens. For buyers, the window size on the spec sheet is the capacity number; effective recall is the spec that decides what you actually get. Summary: on 2025-07-14, Chroma measured all 18 tested models — including GPT-4.1, Claude Opus 4 and Gemini 2.5 Pro — scoring better on LongMemEval with roughly 300 focused tokens than with roughly 113,000 full-context tokens, which is context rot in one number.