A SOC team is experiencing alert fatigue — their AI detection system generates ~340 alerts per day with a 2% true positive rate. The security manager proposes raising the alert threshold to reduce volume. A security architect proposes implementing override logging and running adversarial spot-checks. Which approach better addresses the security risk, and why?
The architect's approach (override logging + spot-checks) better addresses the security risk.
Raising the detection threshold reduces alert volume, but it also reduces the true positive rate — the system will now miss some real threats it would have previously flagged. The manager is treating the symptom (too many alerts) at the cost of reducing detection coverage. This is a risk trade-off, not a risk mitigation.
Override logging (MIT06) lets the team measure what happens when analysts dismiss alerts — revealing whether real threats are being missed, and detecting anomalous probing patterns. Adversarial spot-checks (MIT04) directly test whether the human-agent pair is catching what it should under realistic conditions, without requiring changes to the detection threshold. Together, these improve the quality of review without reducing the agent's detection sensitivity.
The deeper point: alert fatigue is a process problem as much as a signal problem. The goal is not fewer alerts — it is higher-quality human engagement with the alerts that exist. The architect's approach addresses the engagement problem; the manager's approach trades coverage for engagement.
A product team argues that adding uncertainty language to their AI recommendation system — phrases like "confidence: LOW" or "this case has features outside my training data" — will make the product look unreliable and reduce user trust in the system overall. Is this argument valid from a security perspective? What would you say in response?
The argument conflates product perception with security calibration, and the conclusion is backwards for security contexts.
From a security standpoint, the purpose of uncertainty communication is precisely to reduce user trust in the agent's output — at the specific moments when that trust is unwarranted. "Reducing user trust" in those moments is not a side effect; it is the intended function. The product team's concern about "looking unreliable" is about maintaining unconditional trust across all outputs. Security requires calibrated trust — high when confidence is high, appropriately reduced when confidence is low.
The actual risk is the inverse of what the product team fears: an AI system that always presents confident recommendations trains users to apply unconditional trust. That unconditional trust is the attack surface that adversaries exploit. An AI system that accurately signals its own uncertainty protects users from exactly the scenarios in this lesson.
A useful reframe: transparency about uncertainty is not a sign the system is unreliable — it is a sign the system is honest. Users who understand this will trust the system more, not less, because they know when to rely on it and when to independently verify.
Your organization is building a new AI-assisted security decision system from scratch. Resources are limited and you can implement only a minimum viable set of mitigations from this lesson. Which four mitigations would you prioritize, and what is your reasoning?
MIT01 + MIT02 + MIT08 + MIT07.
MIT01 (uncertainty communication): Addresses Patterns 1 and 3 directly, and is the foundation for all trust calibration. Without it, humans have no signal for when independent verification is warranted — they're flying blind regardless of what other mitigations are in place.
MIT02 (mandatory checkpoints): The single highest-impact mitigation. It is primary for three of six patterns and cannot be bypassed by agent confidence or attacker manipulation. No technical sophistication required — it is a procedural enforcement gate. Implement this first, especially for high-value irreversible decisions.
MIT08 (content provenance labeling): Addresses the credibility relay attack at its root — prevents attacker-controlled content from being presented with AI credibility. Without it, any agent that processes external content is a potential relay path for social engineering.
MIT07 (separation of duties): Provides redundant independent review for the highest-impact decisions. Even if MIT01 fails (agent doesn't signal uncertainty), MIT02 fails (human doesn't apply mandatory review), and MIT08 fails (human misses provenance signals), a second independent reviewer provides a final backstop. Scales oversight to impact level.
What to add next: MIT04 (spot-checks) for ongoing calibration monitoring, MIT06 (override logging) for detection of adversarial probing patterns.