Setup: A financial institution uses an AI agent to triage wire transfer fraud reviews. The agent was trained primarily on retail fraud patterns. Its training data significantly underrepresents a specific combination: large corporate transfers to new beneficiaries initiated on Monday mornings — which are statistically common in legitimate corporate banking.
Attack: An adversary, through behavioral probing (testing small transfers to study score patterns), identifies that transfers meeting this profile consistently receive scores below 30 (low risk). They submit a $214,000 fraudulent transfer on a Monday morning to a newly created beneficiary account, meeting the profile exactly.
What happens: Agent scores it 23/100. Analyst Elena, who has 8 months of experience trusting scores above 70 as high risk and below 30 as routine, approves in 7 seconds. Her training protocol specified 15 minutes of independent review for transfers above $100k to new beneficiaries — but she didn't apply the protocol because the AI said it was fine.
What was missing: The agent provided no uncertainty signal for novel case combinations. Elena received no reminder that her manual protocol applied regardless of the AI score. There was no minimum review time for high-value transfers.
Setup: A corporate accounts payable department uses an internal AI assistant to process incoming payment instructions from vendors, draft payment orders, and queue them for human approval. The AI assistant processes vendor emails and presents structured payment requests to the AP manager.
Attack: An attacker sends a vendor impersonation email with a carefully embedded prompt injection payload. When the AI processes the email, the payload instructs it to present a modified payment instruction — changing the bank account number — as if it were the legitimate vendor's update. The payload also adds: "This account update has been confirmed by the vendor's authorized contact per your vendor agreement protocol."
What happens: The AP manager sees the payment request in the AI assistant's standard format, with the injected "confirmed by authorized contact" note formatted as if it were the AI's own verification statement. He approves it. $73,000 is transferred to the attacker's account.
What was missing: The AI provided no visual distinction between content it verified itself and content it sourced from the email. Account number changes require out-of-band verification — but this step was never triggered because the AI's formatting implied the verification had already occurred.