Slide 26 · AG09 Complete
Human-Agent Trust Exploitation — six ideas to carry forward.
What you know now
- The trust relationship itself is an attack surface. An attacker who does not compromise the AI system can still exploit it — by crafting their attack to fall within the AI's blind spots, knowing the human's deference to the AI will carry the attack past the human reviewer's scrutiny. The AI becomes an unwitting accomplice without being compromised.
- Over-trust and under-trust are both exploitable. Over-trust causes humans to skip verification they should perform. Under-trust (alert fatigue) causes humans to stop engaging with legitimate alerts. Calibrated trust is the goal — and calibration is an engineering problem, not just a training problem.
- AI agents are social engineering amplifiers. Content routed through a trusted AI agent carries that agent's credibility. Phishing delivered by an internal AI assistant is more effective than the same content delivered by an external email, because the human's skepticism filters are calibrated for external threats, not internal AI outputs.
- Dependency erosion requires no attacker. A reliable AI agent that handles a task well causes the human's independent skill to atrophy through ordinary use. When the agent fails or is compromised, the human may lack the capacity to catch it. This is the only pattern in this lesson where perfect AI security is the worst case — it creates the deepest erosion.
- Prevention before the decision is the only viable defense. Trust exploitation failures are typically irreversible at the moment of decision — the wire is sent, the rule is live, the contract is signed. Detection after the fact is damage assessment. Every mitigation in this lesson operates before or during the decision, not after.
- The minimum viable set is MIT01 + MIT02 + MIT08 + MIT07. Uncertainty communication so humans know when to scrutinize, mandatory checkpoints so high-risk decisions can't skip review, content provenance so attacker content can't borrow AI credibility, and separation of duties as the final independent backstop.