Slide 12 of 28
Part 2 — Exploitation PatternsSlide 12
Slide 12 · Patterns 5 & 6
Pattern 5: Dependency Erosion — Pattern 6: Oversight Bypass via Urgency Framing
Pattern 5 — Dependency Erosion Long-term degradation

What it is: Over months or years of working with an AI agent, a human professional's independent analytical skills and verification habits gradually atrophy — because they are rarely used. This is a natural consequence of effective AI assistance: if the agent handles a task reliably, the human stops practicing the underlying skill.

The security implication: If the agent is compromised, manipulated, or encounters a case outside its competence, the human who would ordinarily catch the error has lost the skill to do so. Dependency erosion converts the human from a backup check to a rubber stamp — not through any attack on the human, but through the ordinary operation of the AI system over time.

Examples:

  • A security analyst who has relied on an AI threat-triage system for two years no longer has a practiced instinct for which indicators matter — they've outsourced that judgment to the agent.
  • A financial reviewer using an AI contract analysis tool loses the ability to spot non-standard clauses independently, because the agent always caught them — until the one time it didn't.

This is a system design problem: An agent that fully replaces human judgment, rather than augmenting and exercising it, creates this vulnerability as a side effect of doing its job well. Defense requires deliberate design to maintain human skill and verification capability over time.

Pattern 6 — Oversight Bypass via Urgency Framing Procedure bypass

What it is: An agent frames a decision as requiring immediate action, causing the human to skip the deliberate oversight steps that the decision's risk level warrants. The urgency may be fabricated (by an attacker who compromised the agent) or systemic (the agent is designed to prioritize speed over deliberation).

The structural version (more common than the attacker version): Many AI systems are designed to surface high-priority items and resolve them quickly. If the system rewards speed — showing "time in queue" metrics, expressing urgency in framing, or providing approval as the default path — it structurally trains humans to minimize deliberation time. Over time, the review process degrades from genuine evaluation to rapid approval.

The attacker version: An attacker who has compromised or can influence the agent's output can cause the agent to mark any specific decision as urgent — triggering expedited processing, bypassing secondary review queues, and suppressing the human's own sense that the decision warrants more time. This is particularly powerful because the human's response to urgency is automatic: urgency signals activate a different cognitive mode, one that deprioritizes systematic evaluation in favor of fast response.

← Back Scenarios →