AG06:2026 — Memory & Context Poisoning

Slide 27 · Appendix — Real-World Research Context

The research that established memory poisoning as a real attack class — not a theoretical one.

Indirect Prompt Injection — Greshake et al. (2023)

The foundational academic paper establishing indirect prompt injection as a distinct attack class. The research demonstrated that LLMs integrated with external data sources (search engines, documents, APIs) can be manipulated through content those sources return — not through direct user input. The paper included demonstrations against Bing Chat (now Copilot) where injected instructions in web pages retrieved by the model caused it to exfiltrate user data and perform unintended actions.

Relevance to AG06: Established the theoretical and empirical basis for Type 4 (indirect ingestion poisoning) and Type 1 (RAG poisoning via crawled web content) attacks. This work preceded most enterprise RAG deployments.

ChatGPT Long-Term Memory Abuse — Johann Rehberger (2024)

Security researcher Johann Rehberger demonstrated in 2024 that ChatGPT's persistent memory feature — which stores user preferences and facts across conversations — could be abused via indirect prompt injection. A malicious document or web page processed by the model caused it to write attacker-specified facts to its long-term memory store, which then persisted across all future sessions. OpenAI acknowledged and partially patched the issue.

Relevance to AG06: The first public demonstration of Type 3 (preference/profile poisoning) against a production AI system with persistent memory. Demonstrated that the attack survives session boundaries and affects future interactions — the core AG06 harm model.

Poisoned RAG via Document Injection — Multiple Researchers (2024–2025)

Multiple research groups demonstrated that enterprise RAG systems — which automatically ingest uploaded documents, wiki articles, and indexed web content — can be poisoned by inserting malicious documents designed to score high semantic similarity for target queries. Retrieved during legitimate user queries, these documents inject attacker-controlled context into the model's prompt as if it were authoritative knowledge.

Relevance to AG06: Empirical confirmation of Type 1 at enterprise scale. Particularly significant because many organizations deploy RAG systems with automated ingestion pipelines that process untrusted external content alongside trusted internal documents — with no clear separation between the two.

← Back Sources →