Slide 11 of 27
Part 2 · TypesSlide 11
Slide 11 · Multi-Step Extraction — Real Incident
GitHub Copilot Chat System Prompt Extracted — May 2023.
Confirmed Incident · May 2023 · Disclosed on Hacker News & GitHub
Researchers Extract GitHub Copilot’s Full System Prompt via Multi-Turn Probing
No CVE · Multi-step indirect extraction · Archived publicly on GitHub; discussed on Hacker News May 2023

The setup: GitHub Copilot Chat ships with a system prompt that defines its available tools (code execution, file access, search), its constraints, and its operational context. It was not disclosed to users.

The extraction: Researchers used a sequence of probing questions across multiple conversation turns — asking about Copilot’s capabilities, what tools it had access to, what it wasn’t allowed to do — and assembled the full prompt from the model’s responses. Unlike Bing, no single instruction broke the secrecy; it was patience and inference.

The result: The extracted prompt was posted to GitHub repositories and archived. It revealed Copilot’s internal tool list, function signatures, and behavioral constraints in full detail.

Why it matters for LLM07: Multi-step extraction works even when a model resists direct requests. Each answer about what the model “can” and “can’t” do narrows the space. Given enough turns, the prompt is reconstructed from fragments.
The Defense That Would Have Worked

Keeping tool descriptions and function signatures out of the prompt (using backend code instead) would have made the extracted content useless. The architecture was the vulnerability — not the lack of a better secrecy instruction.

← BackNext → Encoding-based extraction