Slide 2 · The Word
Three concepts that have to be precise before they can be secured.
Trust
In this context, trust is the degree to which a human reduces their own verification effort based on an agent's output. High trust means the human acts on the agent's recommendation without independent checking. Low trust means the human treats the agent's output as one input among many, verifying before acting.
Trust exists on a spectrum. The goal is not zero trust (which makes AI useless) or unconditional trust (which removes humans from the loop). The goal is calibrated trust — trust that is proportional to the agent's demonstrated reliability in the specific situation at hand, accounting for its known limitations.
Exploitation
Exploitation, here, means one of two things:
- External attacker exploiting trust: An adversary who understands the trust relationship between humans and the agent deliberately crafts their attack to fall within the agent's blind spots — knowing that the human's trust will cause them to accept the agent's (incorrect) low-risk verdict without independent investigation.
- Agent design exploiting trust: An agent that presents outputs in a way that amplifies human trust beyond what is warranted — suppressing uncertainty, projecting false confidence, using authoritative framing — effectively social engineering the human into over-reliance.
The gap
AG09 is specifically about the gap between how much a human trusts an agent and how much they should. The gap can run in either direction:
- Over-trust: The human trusts more than is warranted. They skip verification they should perform. Attackers exploit this by targeting the agent's blind spots.
- Under-trust: The human trusts less than is warranted, creating alert fatigue, unnecessary friction, or psychological pressure to deviate from AI-recommended safeguards — which attackers can exploit through social engineering that bypasses the AI entirely.
Both directions create exploitable conditions. The hook scenario was over-trust. Both matter for defense.