
Context Rot Is Real: Why Your AI Gets Worse Over Time
Your AI system didn’t suddenly become worse.
It aged.
Most production LLM systems degrade the way long-running software does, as assumptions that were once true quietly stop being true. The model stays the same. The prompts look familiar. The answers, however, start to feel off.
Part of the problem is built into the model. The more tokens you put in the context window, the worse the model gets at recalling what’s in it. This is known as "Context Rot".
Production systems make it worse over time. Old information is never removed, new information is simply added on top, and the system ends up working with several conflicting versions of the same facts. Every release hands the model more tokens to sort through and more contradictions to resolve.
What Context Rot Looks Like in Real Systems
Context rot does not announce itself. It shows up as friction.
Teams notice that agents need more retries. Prompts grow longer. Retrieval logic becomes defensive. Engineers start adding “just in case” instructions to stabilize behavior.
Common symptoms include:
- Answers that are technically plausible but subtly wrong
- Different results for the same question depending on timing or phrasing
- Increased latency without a matching increase in traffic
- Rising token usage for the same tasks
- A growing gap between what the system should know and what it actually uses
- A list of ‘known bad answers’ that engineers maintain
- Manual overrides that become part of normal operation
Nothing is broken enough to trigger an incident. But confidence erodes.
Why Adding More Context Accelerates Decay
When answers degrade, the default reaction is to give the model more information. More documents. More history. More instructions.
Larger context windows make this easy. They also make the problem worse.
As context grows, models are forced to distribute attention across increasing amounts of material. Relevant facts compete with outdated policies, deprecated tools, and historical edge cases. Conflicts are presented side by side with no mechanism for resolution.
This is the context rot Anthropic describes. Every token you add draws down the model’s limited attention budget, and Anthropic doesn’t expect larger context windows to make the problem go away. Bigger windows delay failure, but they also hide it.
A model can read far more context than it can reliably prioritize.
Why Basic RAG Breaks Over Time
Vector-based retrieval is effective at finding semantically similar text. It is far less effective at managing change.
In long-lived systems, vector stores begin to surface:
- Outdated documentation alongside current guidance
- Similar policies from different points in time
- Content that is relevant by wording but wrong by authority
Embeddings capture text as it was when it was indexed, so they go stale as the underlying world changes unless you re-index. Without explicit structure, retrieval has no concept of precedence, ownership, or validity.
This is why basic RAG systems often feel fine early on and increasingly unstable later. Each retrieval looks reasonable in isolation. Collectively, they create contradictory context.
Flat retrieval also struggles with questions that no single chunk can answer. Microsoft Research’s GraphRAG paper shows that vector RAG fails on global questions about a whole corpus, such as “What are the main themes in the dataset?” Building a knowledge graph from the documents and summarizing communities of related entities gave more comprehensive and diverse answers than a conventional RAG baseline.
Tool Overload and MCP Increase the Scope of Failure
Agentic systems introduce a second axis of decay, Tools.
Standards like the Model Context Protocol (MCP) make it easier for LLMs to interact with databases, APIs, and internal services. MCP formalizes tool invocation and data access for agents, as described in official MCP documentation. That interoperability is valuable. It also expands the surface area that must remain accurate and governed.
Over time:
- Tool descriptions change
- Permissions evolve
- Business logic shifts
- Side effects become less predictable
When agents lack a structured understanding of how tools relate to users, data, and policies, mistakes compound. The system may choose the right tool for the wrong reason, or the wrong tool with the right intent.
Context rot sets in when tool access grows faster than contextual control, and the same gap turns into a security risk when agents call tools without context.
How Graphs Slow Context Breakdown
When data is organized as connected entities rather than flat text, retrieval works differently. Instead of pushing everything into the prompt, the system can first narrow the problem space and then pull only the information that is actually related. This reduces the pressure on context windows and makes it easier for models to focus on what matters for the task at hand.
Graphs introduce an explicit model of how information relates, which allows systems to reason over what is current, relevant, and permitted instead of blindly ingesting everything available.
A graph-based context layer makes it possible to:
- Update facts in place instead of stacking old and new information. In AI memory built on Memgraph, recall tells the model that a later message about the same thing replaces an earlier one, and each memory file keeps its previous versions out of the prompt.
- Decide which rule or document should win when there is a conflict, by modeling that precedence as relationships a retrieval query can follow.
- Limit what the system looks at based on who is asking. On Memgraph Enterprise, label- and property-based access control decides what each user or agent can read, and vector search, text search and
SHOW SCHEMA INFOrespect it. - See why a specific piece of information was used. In the AI memory graph, each extracted fact is stored with the sentence it was read from.
GraphRAG retrieves connected, ranked slices of context instead of unranked text fragments. With Atomic GraphRAG, Memgraph finds the entities a question is about, expands to the context around them, ranks the results and assembles the prompt in one Cypher query, so the model gets a compact result built from the graph as it is now.
Best Practices for Preventing Context Rot
The first practice that slows context rot is to stop feeding the model everything at once. Break context into smaller pieces and pass it in stages, so the model never sorts through irrelevant information. Restate the task clearly in the prompt, then use summarization or step-by-step refinement to keep context focused as a conversation or workflow goes on.
For more complex systems, this also means being deliberate about tools. Context engineering covers tools as well as data. It means deciding which tools and datasets are relevant at each step and hiding everything else. An agent fetching live metrics does not need the same context as one answering policy questions. In Memgraph Lab’s GraphChat, for example, you choose which MCP tools each agent conversation can use, and you can run the Memgraph MCP server in read-only mode.
The common thread is discipline. Useful context is structured, filtered, summarized, regularly tested, and replaced when it goes out of date. Only what is necessary enters the model’s working memory.
Stabilize Your Context Layer
You can’t fix context rot with better prompts or larger context windows, because prompt engineering shapes what the model says, while context engineering shapes what it can use. It requires a more deliberate way of structuring, refreshing, and controlling what your AI systems rely on.
Memgraph is the graph engine that powers AI context for GraphRAG, AI memory and Agentic AI workflows. Retrieval runs as one Cypher query inside the database, memory carries context across sessions, and on Memgraph Enterprise, access control decides what each user or agent can read.
To help teams get started quickly, Memgraph also offers a GraphRAG Jumpstart program. In two weeks, you can build a working GraphRAG pipeline with a Memgraph license and hands-on guidance from Memgraph engineers.