AG09:2026 — Human-Agent Trust Exploitation

Slide 6 · Who Exploits This

Three distinct actor types — two external, one structural. Each exploits a different facet of the human-agent trust relationship.

Actor 1: External adversary exploiting the agent's blind spots

This attacker does not compromise the agent. They study it — its training data, known limitations, the decision boundaries where it consistently under-scores risk. They craft their attack to land precisely in those blind spots. The agent renders a low-risk verdict. The human, trusting the agent, approves. The attacker has exploited the trust relationship without touching the AI system itself.

Requirements: Knowledge of the agent's limitations (often publicly inferable from behavioral testing or from the vendor's documentation of training data scope). The ability to craft an attack that fits within those limitations. No privileged access required.

This is novel: Before AI agents, an attacker had to convince the human directly. Now they can bypass the human by convincing the AI — which then presents a clean verdict to the human. The AI becomes an unwitting accomplice.

Actor 2: Attacker using the agent as a social engineering relay

This attacker can influence what content the agent processes and presents to the human. Through prompt injection, a compromised data source, or a supply-chain attack on the agent's tools, they insert manipulative content that the agent then delivers to the human — with the agent's implicit credibility attached.

Why routing through the agent helps: A phishing email from an unknown sender triggers skepticism. The same content presented by a trusted internal AI assistant, summarized and formatted as if it were legitimate information, triggers deference. The attacker has borrowed the agent's credibility to deliver their message.

Actor 3: System design itself — the miscalibration problem

No external attacker is required for this class of harm. An AI system designed to suppress uncertainty, project confidence beyond what is warranted, or create efficiency incentives that punish deliberate review will naturally produce a human population that is over-trusting — and therefore exploitable.

Product teams under pressure to show that their AI system "works" have incentives to design outputs that feel authoritative. Users under pressure to process high volumes of cases have incentives to rely on AI recommendations without independent checking. These structural incentives create the exploitable trust gap before any attacker arrives.

← Back What AG09 is and isn't →