Slide 16 of 27
Part 3 · ScenariosSlide 16
Slide 16 · Scenario 3 of 3
ConfusedPilot — one document, every Copilot user misled.
Research Demonstration · November 2024 · UT Austin SPARK Lab & Symmetry Systems
ConfusedPilot: Confused Deputy Risks in RAG-Based AI Systems
No CVE · Disclosed at Cloud Security Alliance · November 12, 2024

The setup: Microsoft 365 Copilot and all major RAG-based enterprise AI systems index documents from the environment. An attacker with minimal access — enough to save one document to any indexed location — can manipulate AI responses organization-wide.

The attack: The attacker places a document containing specially crafted strings in any indexed folder. When any user queries Copilot on a related topic, the RAG system retrieves the document and the LLM interprets its content as instructions — suppressing legitimate information, generating false claims, or misattributing sources.

The persistence problem: Responses remain manipulated even after the malicious document is deleted. The embedding persists in the vector cache until the system reindexes, continuing to influence AI responses with no document present to investigate.

Scope: Researchers noted that 65% of Fortune 500 companies were adopting or planning RAG-based AI at the time of disclosure — making this a broad-surface risk, not a niche one.

Why it matters: Minimal access, organization-wide impact, and persistence after cleanup. ConfusedPilot demonstrates that LLM08 applies to the exact enterprise AI tools organizations are already running.
← BackNext → The Pattern