Slide 2 of 28
Part 1 — The ProblemSlide 2
Slide 2 · The Word
Three concepts that have to be precise before they can be secured.
Trust

In this context, trust is the degree to which a human reduces their own verification effort based on an agent's output. High trust means the human acts on the agent's recommendation without independent checking. Low trust means the human treats the agent's output as one input among many, verifying before acting.

Trust exists on a spectrum. The goal is not zero trust (which makes AI useless) or unconditional trust (which removes humans from the loop). The goal is calibrated trust — trust that is proportional to the agent's demonstrated reliability in the specific situation at hand, accounting for its known limitations.

Exploitation

Exploitation, here, means one of two things:

The gap

AG09 is specifically about the gap between how much a human trusts an agent and how much they should. The gap can run in either direction:

Both directions create exploitable conditions. The hook scenario was over-trust. Both matter for defense.

← Back Official definition →