Context Rot in Long-Context LLMs
Why longer prompts make models worse at finding the right code, and how coding agents avoid look-alike distractors.
Model vendors now sell context windows of 128k to 1M+ tokens. The context window is how much text the model can read at once, counted in tokens (small chunks of text, often parts of words). The usual assumption is that the model reads all of that space equally well. A rule at token 100k should be as easy to recall as a rule at token 100.
It is not. Chroma tested this in Context Rot. As the input gets longer, the model gets worse at finding the right fact. It gets much worse when the question and the answer use different words. Chroma calls this context rot: the longer the prompt, the less reliably the model uses what is in it.
The Uniformity Myth
Models use self-attention to decide which earlier tokens matter for the next one. In theory, every token can attend to every other token. In practice, the model has a fixed amount of attention to spread, and a long input spreads it thin. When an agent dumps whole directories into a prompt, most of that attention lands on code that does not matter. You get three problems:
- Instruction drift: The model stops following rules from the start of the prompt, such as the system prompt.
- Attribution blending: The model merges pieces of unrelated files into one implementation that does not work.
- Hallucination: The model cannot find a detail in the pile, so it invents one.
Semantic Degradation vs. Standard NIAH
The standard long-context test is Needle in a Haystack (NIAH). It hides one sentence (the needle) in a long block of unrelated text (the haystack) and asks the model to find it. The question and the needle usually share the exact same words. This is lexical matching: matching on identical words. It is basically grep with extra steps, and models score well on it. Those scores flatter them.
Chroma also tested semantic, non-lexical matching. Here the question and the answer mean the same thing but share no keywords. On this test, accuracy drops fast as the input grows:
Retrieval Accuracy
100% |===============\
| \ Lexical Matching (Standard NIAH)
50% | \ \---------------------------
| \--- Semantic Non-Lexical Matching
0% +---------------------------------------------
1K 16K 64K 128K 256K+
Input Tokens
Real code questions look like the bottom line, not the top one. You rarely ask an agent for a function by its exact name. You ask it to “fix the login bug”.
Distractors in Codebases
Chroma separates two kinds of extra text in a prompt:
- Noise: Text unrelated to the question, such as history essays mixed into documentation. Models ignore noise fairly well.
- Distractors: Text on the same topic that looks like the answer but is wrong.
A codebase is mostly distractors. You have overloaded signatures (several functions with near-identical names and shapes, like getUser and getUserById). You have old legacy modules nobody deleted. You have test mocks that copy the real interfaces. If an agent loads 50 files without filtering them, the model often picks the wrong function or imports a deprecated helper. It is not being careless. You handed it five plausible answers and asked it to pick one.
How December Mitigates Context Rot
December does not dump the repo into the prompt. It loads only what the task needs:
- Localized AST extraction: December parses your code into an AST (abstract syntax tree, a structured map of every symbol in a file). It loads only the symbols and type interfaces the task uses, not whole files.
- Pruned agent reasoning loop: Once December verifies a step, it drops that step’s terminal output and planning notes from context.
- Deterministic checkpoints: December checks its work with signals that give the same answer every time: compiler errors and test exit codes, run in an isolated sandbox. It does not trust what the prompt “remembers”.
The short version: a bigger context window is a bigger haystack. It is not a better search.
Naive pipeline
Dumps the whole repo: 128K+ tokens of distractors. Result: wrong imports and instruction drift.
December pruned pipeline
Extracts AST symbols: about 4K tokens. Result: edits checked by the compiler and tests.
Reference: Chroma Technical Report: Context Rot.