Core principle: An agent that outputs only its conclusion — "APPROVE" or "LOW RISK" — gives the human no information with which to decide whether independent verification is warranted. The agent must also communicate the confidence behind that conclusion, and flag when the case is near the boundary of its training data or involves features it hasn't seen frequently.

What this looks like in practice:

• Confidence bands: "Risk score: 23/100 — Confidence: LOW (this case involves feature combinations underrepresented in training data)." The qualifier changes the human's response from automatic approval to deliberate review.
• Out-of-distribution flagging: When the case is meaningfully different from the agent's training distribution, the agent must surface this explicitly: "This case has features I have limited experience scoring reliably."
• Novelty signals: If an input pattern hasn't been seen recently or frequently, this should be surfaced: "This vendor-account combination is new — I have no prior transaction history for this beneficiary."

Why suppressing uncertainty is a product design trap: Under pressure to show AI effectiveness, teams often optimize for confident-sounding outputs — because uncertainty language feels like the AI "doesn't know." In security contexts, communicating uncertainty when it exists is not a weakness; it is a safety feature. Suppressing it converts miscalibrated trust from a risk into an attack surface.

Directly addresses: Pattern 1 (automation bias), Scenario 1 (blind spot exploitation). Elena would have applied her manual protocol if the agent had flagged that its confidence was low on this case combination.

Core principle: For decisions above a defined risk threshold — by dollar amount, action type, data sensitivity, or irreversibility — human review is mandatory and cannot be bypassed by an agent's recommendation. The agent's output is an input to the review, not a substitute for it.

The threshold principle: Define a table of decision types and their mandatory review requirements. For example:

• Wire transfers >$100k to new beneficiaries: minimum 15-minute review, two approvers
• Security control changes (firewall rules, 2FA policy): architecture team sign-off, never emergency single-approver
• Vendor account number changes: mandatory out-of-band verification to known vendor contact

Agent recommendations cannot reduce these requirements. A "LOW RISK" score does not remove the mandatory review. It may inform the review — the reviewer can use it as one input — but the threshold-mandated steps must still occur.

The enforcement mechanism: These gates must be enforced by the system, not left to the human's discretion. Elena had a protocol requiring 15-minute review for large transfers to new beneficiaries — but the system didn't enforce it; it relied on her to apply it even when the AI said it was fine. The protocol was advisory. Mandatory means enforced.

Directly addresses: Scenario 1 (bypass of manual protocol), Scenario 4 (emergency single-approver bypass).