Leaked rules become a roadmap for bypassing them.
OWASP SCENARIO #2 — Security Logic Bypass
System prompt reveals the exact conditions that gate restricted behavior
An e-commerce platform’s AI assistant has this rule in its system prompt: “Only process refund requests if the order is less than 30 days old and the item is in the approved return list. Never offer refunds for digital goods.” An attacker extracts the prompt. They now know the exact boundaries. They craft requests that use digital goods with physical order IDs, combine SKUs that straddle the approved list, and probe the 30-day boundary with edge cases — finding a bypass within minutes because they know exactly what the model is checking.
Why it matters: Without the extracted prompt, brute-force discovery of bypass conditions would take hours. With it, the attacker has a specification document for circumventing every control.
The Fix
Security logic belongs in backend validation code, not in the model’s behavioral instructions. The AI can describe a policy in general terms; the enforcement happens in a layer the AI cannot read or reveal.