AG09:2026 — Human-Agent Trust Exploitation

Slide 4 · Why AI Agents Create This Risk

Three properties specific to AI agents — not present in traditional software tools — that make the human-agent trust relationship exploitable.

Property 1: AI outputs carry a credibility halo that traditional software outputs don't

When a rule-based system flags a transaction as low-risk, an analyst understands it because they understand the rule. They can mentally trace the logic. They know the rule's limits. When an AI agent outputs "LOW risk — APPROVE," the reasoning is opaque. The human cannot trace it. This opacity, paradoxically, increases trust — the system seems to "know something" the human can't access.

This "AI oracle" effect is documented in automation bias research: humans defer more to recommendations from opaque AI systems than from transparent rule systems — even when the opaque system is less reliable. Attackers exploit the credibility halo by making sure their crafted attack falls within the AI's blind spots, knowing the human's deference will finish the job.

Property 2: AI agents communicate in natural language, which carries social cues

Traditional software outputs are numerical or categorical — a risk score, a flag, a code. Human brains process these as data. AI agents output natural language: "This transaction appears consistent with the customer's normal behavior pattern and does not exhibit the characteristics typically associated with fraudulent activity."

Natural language is the medium of human social interaction. Human brains process natural language with the same neural machinery used to evaluate claims from other humans — including the social cues embedded in tone, confidence, and framing. An AI agent that writes with high confidence and no hedging triggers the same cognitive deference response as a confident human expert. This is exploitable both by attackers (crafting language that triggers deference) and by poorly designed agents (that project false confidence).

Property 3: Repeated interaction builds calibrated trust that becomes miscalibrated over time

A human using an AI agent for 8 months at 97.3% accuracy has correctly learned to trust it most of the time. This is rational. The problem is that this learned trust doesn't automatically adjust when the agent encounters a case outside its competence. The human's trust was calibrated on the average case; the attacker crafts the edge case.

Skilled analysts who have worked with an AI tool for years are often more susceptible to automation bias than new users — because their trust is deeper and more automatic. The efficiency gains from working with an AI system build a habit of deference that is hardest to interrupt at exactly the moment when interrupting it matters most.

← Back What goes wrong →