Slide 20 of 28
Part 3 — PreventionSlide 20
Slide 20 · MIT05 & MIT06
MIT05: Trust Calibration Training — MIT06: Override Logging and Review
MIT05 — Trust Calibration Training

Core principle: Generic "don't trust AI too much" training does not help humans in the moment when they have 7 seconds to review a transaction that the agent scored as low risk. Effective trust calibration training is specific: it teaches users exactly what the agent's limitations are, what case categories are most likely to fall in its blind spots, and what verification steps are most important when those cases appear.

What makes this training different from generic security awareness training:

The alert fatigue angle: MIT05 also addresses Pattern 3 — teams should be trained on alert quality metrics, how to recognize when their agent is generating noise, and how to request recalibration. Passive alert fatigue is a training problem as much as a system problem.

Directly addresses: Pattern 1 (Elena didn't know which case categories were outside the agent's reliable range), Pattern 3 (SOC analysts didn't recognize their alert false-positive rate as a signal to escalate).

MIT06 — Override Logging and Review

Core principle: Overrides — cases where a human decides differently from the agent's recommendation — are a gold mine of trust calibration signal. Systematic review of overrides reveals: where the agent is wrong (humans override it and they're right), where humans are over-trusting (agent is right but humans sometimes ignore it), and where adversarial probing patterns may be emerging (unusual override patterns may be an attacker testing boundaries).

Log both directions:

Adversarial probing detection: An attacker systematically testing an agent's boundaries to find its blind spots will generate an unusual pattern of low-risk cases that are genuinely high risk. If a reviewer repeatedly escalates agent-approved cases from a specific source, category, or time window, this pattern is detectable in override logs before any successful attack completes. MIT06 is also an early warning system.

Directly addresses: Pattern 1 (would have revealed the blind spot being probed in Scenario 1), Pattern 3 (override logs would reveal systematic alert dismissal patterns).

💼 Business takeaway

Ask your team: when a reviewer disagrees with the AI's recommendation, is that disagreement recorded anywhere? Ask whether a pattern of overrides — reviewers repeatedly escalating cases the agent approved — would be noticed before it became a security incident.

← Back MIT07, MIT08 & MIT09 →