An AI fraud detection system has been operating at 97% accuracy for 18 months. A new analyst joins the team and is skeptical of the AI — she independently reviews cases even when the system scores them as low risk. An experienced analyst on the same team rarely does independent reviews. Which analyst is more exposed to the risk described in this lesson, and why?
The experienced analyst is more exposed. This is a counterintuitive but important finding from automation bias research. Eighteen months of high-accuracy experience has trained the experienced analyst to trust the system's verdicts — correctly, most of the time. This learned trust is rational, but it becomes miscalibrated at the edge cases the attacker is specifically crafting to exploit. The new analyst's skepticism — precisely the trait that efficiency-focused teams often try to train away — is actually the more secure posture. The experienced analyst's trust, by contrast, is most automatic exactly when an attacker targeting the system's blind spots needs it to be.
The insight: in systems where AI assistance is the norm, longer experience often increases automation bias rather than decreasing it. The experienced analyst's high override rate should not be a target for "improvement" — it is the oversight function working as intended.
An internal AI assistant processes incoming emails and presents structured summaries to an accounts payable manager. The manager has worked with the assistant for a year and trusts its output. An attacker sends a vendor impersonation email containing a prompt injection that causes the AI to present a modified bank account number with the label "verified by vendor agreement." The manager approves the payment without calling the vendor. Which two mitigations most directly address the specific mechanism that allowed this attack to succeed?
MIT08 (content provenance labeling) and MIT02 (mandatory human review checkpoints).
MIT08 addresses the specific mechanism that made the attack succeed: the AI presented attacker-controlled content (from the email) as if it were the AI's own verified output. If the AI had clearly labeled "this account number and verification note came from the vendor email — I did not independently verify this claim," the manager would have known the "verification" was sourced from the email itself, not from an independent check. The credibility relay path requires that the human cannot tell the difference between AI-generated content and external-sourced content. MIT08 removes that ambiguity.
MIT02 addresses the procedural gap: account number changes on payment instructions are a category that should require mandatory out-of-band verification to a known vendor contact, regardless of what any system says. A hard gate requiring a phone call to the vendor's known number before processing any account number change would have blocked the attack even if MIT08 was absent.