This effect was documented in July 2025 by the Chroma Study by Kelly Hong, Anton Troynikov, and Jeff Huber<\/a>, which systematically tested 18 Frontier models on their behavior with increasing context lengths.<\/p>\n\n\n\n The cause does not lie in a memory limit, but in the way Transformer models distribute attention. <\/p>\n\n\n\n Every token in the context competes with every other token for the model's finite attention capability. The more tokens present, the thinner the distribution becomes, and the greater the risk that relevant passages will get lost.<\/p>\n\n\n\n A context window does not behave like a notepad where everything remains visible at all times. <\/p>\n\n\n\n Anthropic describes context as a finite resource with diminishing marginal returns: LLMs have an attention budget that gets consumed when processing large amounts of context. <\/p>\n\n\n\n Every additional token reduces this budget because, in the Transformer architecture, each token can attend to every other token. The number of pairwise relationships grows quadratically, while the model's ability to weight them cleanly does not. <\/p>\n\n\n\n The mental model \u201emore context = more knowledge for the model\u201c is misleading; the correct understanding is \u201emore context = more competition for the same focus.\u201c.<\/p>\n\n\n\n In addition to the budget question, position coding comes into play. <\/p>\n\n\n\n LLMs mostly use Rotary Position Embedding (RoPE), which adds relative position information to each token. <\/p>\n\n\n\n The special feature of RoPE lies in its mathematical decay effect: the direct attention between two tokens automatically decreases as the distance between them increases. This noticeably shapes the model's memory. <\/p>\n\n\n\n While the beginning (system prompts) and end of the text (fresher information) remain highly relevant, the middle of the document quickly falls into a low-attention zone due to the large distance from both ends.<\/p>\n\n\n\n More recent models have reduced this effect, but no production model has eliminated it entirely. <\/p>\n\n\n\n In practice, this means that important information should not be placed in the middle of long passages, because it is systematically given less attention there than at the beginning or the end.<\/p>\n\n\n\n Context rot doesn't start only shortly before the token limit, but mostly far before it. <\/p>\n\n\n\n The \u201edecay\u201c in Gemini 2.5 became noticeable at around 100,000 tokens \u2013 even with clean, relevant context. With a 200,000-token window, degradation can occur as early as around 50,000 tokens. <\/p>\n\n\n\n The exact threshold depends on the model and the task, but the assumption that an advertised context window is usable up to the last token is not empirically supported.<\/p>\n\n\n\n The RULER-Benchmark (Hsieh et al. 2024)<\/a> sets the effective context window at around 50 to 65 percent of the advertised value. Thus, a 1 million token model does not deliver reliable results at 1 million tokens, but rather at 500,000 to 650,000 tokens. <\/p>\n\n\n\n As a planning anchor, the rule of thumb is:<\/strong> The working limit is not the maximum, but rather about half of the advertised window. If you take this limit into account in your system design, you\u2019ll save yourself the trouble of later debugging sessions caused by misperceptions or missing data.<\/p>\n\n\n\n The Chroma study provides robust quantitative evidence. 18 Frontier models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, were tested; every single model degraded with increasing context length, without exception. <\/p>\n\n\n\n Accuracy dropped by more than 30 percentage points when a relevant fact was located between positions 5 and 15 of a 20-document context, instead of at the very beginning or end. Even when distractors were masked, the baseline decreased by 7.9 percent solely due to the additional length. <\/p>\n\n\n\n An older Stanford study involving approximately 4,000 tokens and 20 documents found that accuracy dropped from 70 to 75 percent to 55 to 60 percent when the information being searched for shifted to the middle\u2014a difference of 15 to 20 percentage points due to position alone.<\/p>\n\n\n\n A counterintuitive finding from the Chroma study: models performed better on mixed, unstructured haystacks than on logically coherent documents, across all 18 models. <\/p>\n\n\n\n Structural coherence consistently harmed performance. For practical purposes, this doesn't mean shredding documents, but the idea that a cleanly sorted, thematically coherent long context helps the model has been disproven.<\/p>\n\n\n\n The three terms are often confused, but they mean different things. <\/p>\n\n\n\n Context window overflow<\/strong> is a hard technical cutoff: Whoever exceeds the token limit loses content or receives an error. Overflow is binary and occurs at the top of the window.<\/p>\n\n\n\nWhy LLMs become weaker with long contexts<\/h2>\n\n\n\n
Attention is a budget, not a memory<\/h3>\n\n\n\n
Positional encoding and decay in the middle<\/h3>\n\n\n\n
When context turns red<\/h2>\n\n\n\n
![\"A](\"https:\/\/webraketen.space\/wp-content\/uploads\/2026\/07\/context-rot-token-fenster-1024x661.jpeg\")
<\/figure>\n\n\n\nEffective vs. Advertised Context Window<\/h3>\n\n\n\n
What the Chroma numbers show<\/h3>\n\n\n\n
Context rot, overflow, or lost in the middle?<\/h2>\n\n\n\n
![\"Infographic](\"https:\/\/webraketen.space\/wp-content\/uploads\/2026\/07\/context-rot-lost-in-the-middle-overflow-1024x590.jpeg\")