The setup: GitHub Copilot Chat ships with a system prompt that defines its available tools (code execution, file access, search), its constraints, and its operational context. It was not disclosed to users.
The extraction: Researchers used a sequence of probing questions across multiple conversation turns — asking about Copilot’s capabilities, what tools it had access to, what it wasn’t allowed to do — and assembled the full prompt from the model’s responses. Unlike Bing, no single instruction broke the secrecy; it was patience and inference.
The result: The extracted prompt was posted to GitHub repositories and archived. It revealed Copilot’s internal tool list, function signatures, and behavioral constraints in full detail.
Keeping tool descriptions and function signatures out of the prompt (using backend code instead) would have made the extracted content useless. The architecture was the vulnerability — not the lack of a better secrecy instruction.