context poisoning
In plain wordsWhen a mistake or a planted falsehood gets into an AI's working notes or memory and the AI keeps relying on it, so the error grows over later steps.
- 1
Agents (failure mode)
A failure mode of long-running agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. in which a mistaken belief, often a hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. about how far the task has progressed, gets recorded in contextcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. material the agent keeps consulting (a goals list, summary, plan or scratchpadscratchpad, n.: A space where an AI writes rough working — calculations, plans, notes — before or while producing its answer, much as a person jots on scrap paper.); every later step then builds on the false record, and the agent may chase goals that cannot be reached or no longer matter. Since nothing in the loop rechecks the record, the error compounds instead of being corrected. Mitigations include validating facts before persisting them, separating verified state from model-generated beliefs, periodically rebuilding context from ground truth, compactioncompaction, n.: Like replacing a stack of rough notes with a one-page digest: when an AI's conversation grows too long for its memory, the history is condensed into a summary and the work continues from there. that drops dubious material, and isolating exploratory work in subagentssubagent, n.: A helper AI agent that a main agent starts to handle one piece of a job; it works in its own separate workspace and reports back a short summary when it is done..
Once the agent wrote a nonexistent API into its plan, context poisoning set in and every later step tried to call it.
- 2
Security
An attack in which an adversary plants misleading content or hidden instructions in material an agent will store and later trust — persistent memory, shared state, retrieved documents or tool results — so that the manipulation outlives the turn in which it entered. It is a persistent form of indirect prompt injectionindirect prompt injection, n.: A prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it. and an inference-time relative of data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.. Defenses include provenance tags and trust levels for stored content, re-screening memories when they are read back, and limiting what agents may write to long-term stores.
Origin
Google DeepMind's Gemini 2.5 technical report (June 2025) used 'context poisoning' for the Gemini Plays Pokémon agent's fixation on hallucinated goals recorded in its context. Drew Breunig listed it first among four ways long contexts fail (22 June 2025), alongside distraction, confusion and clash. The security sense, overlapping with 'memory poisoning' in agent-security threat lists, spread during 2025 with attacks on agent memory and Model Context ProtocolModel Context Protocol, prop. n.: An open standard that lets AI assistants plug into outside apps and data — files, calendars, databases, business software — through one common connector instead of a custom link for every pairing; often likened to a USB-C port for AI. tools.
Usage note
In the first sense nobody attacks the system; the agent poisons itself. Contrast data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., which corrupts training data rather than the context at inference time.
Related
- context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere.
- context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else.
- agent memoryagent memory, n.: The ways an AI assistant or agent keeps information — your preferences, past conversations, lessons from earlier tasks — and brings it back later, beyond what fits in a single conversation.
- hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
- indirect prompt injectionindirect prompt injection, n.: A prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it.
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
- scratchpadscratchpad, n.: A space where an AI writes rough working — calculations, plans, notes — before or while producing its answer, much as a person jots on scrap paper.
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
Sources
- Breunig, D. (2025). How Long Contexts Fail. dbreunig.com, 22 June 2025. dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html
- Comanici, G. et al. (2025). Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv:2507.06261. arxiv.org/abs/2507.06261
Mentioned in
agent memoryagent memory, n.: The ways an AI assistant or agent keeps information — your preferences, past conversations, lessons from earlier tasks — and brings it back later, beyond what fits in a single conversation., context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else., context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere., data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.