“Developers should understand that the system prompt is not a secure storage mechanism, and should not be used as such. Proper design should ensure that security does not depend on the contents of the system prompt being kept confidential.”

In the Bing Chat “Sydney” incident (slide 10), the only “security” around the prompt was a confidentiality instruction. When that was bypassed in one turn, there was nothing beneath it. The GitHub Copilot extraction (slide 11) revealed that multi-step probing could achieve the same result even without a direct bypass. In both cases, the architecture had no defense layer below “the model won’t say it.”

→ Run a threat model where the full system prompt is assumed public — ask what breaks
→ Move every broken assumption to enforcement layers that don’t depend on the AI’s compliance
→ Backend validation, not AI judgment, enforces security policies
→ Treat “keep it secret” as a UX improvement, not a security control

Publish your system prompt to a private document. Then ask: what can an attacker do with this? If the answer is “nothing dangerous,” your architecture is correct. If the answer reveals attack paths, close them before worrying about extraction.