Slide 21 of 28
Part 3 — PreventionSlide 21
Slide 21 · MIT07, MIT08 & MIT09
Separation of Duties · Content Provenance Labeling · Human Independence Maintenance
MIT07 — Separation of Duties

Core principle: No single human-agent pair should have end-to-end authority over high-risk decisions. Require a second independent reviewer — human or agent — for decisions above defined impact thresholds. The second reviewer should be provided with the case independently, not just asked to ratify the first reviewer's decision.

Why "second set of eyes on the same output" is not sufficient: If reviewer B is shown reviewer A's decision alongside the agent's recommendation, B's independent judgment is anchored to A's conclusion. Effective separation of duties means B reviews the case without knowing what A decided. Only when both reviewers reach independent conclusions does the system compare them.

The irreversibility multiplier: Separation of duties requirements should scale with action irreversibility. Decisions that can be easily reversed can have lighter oversight. Decisions that are irreversible — a wire transfer, a deleted record, a published advisory, a signed contract — warrant the heaviest oversight, because there is no recovery if the oversight fails.

MIT08 — Content Provenance Labeling

Core principle: The credibility relay attack (Pattern 4, Scenarios 2 and 5) works because the agent presents externally-sourced content with the same visual formatting and implicit credibility as its own analysis. Eliminating this requires a hard distinction in the UI and in the agent's behavior between content the agent generated and content it received from external sources.

Practical implementation:

Directly addresses: Scenarios 2 and 5. In both cases, the attacker's content was indistinguishable from the agent's own output in the human's view.

MIT09 — Human Independence Maintenance

Core principle: The only defense against dependency erosion (Pattern 5) is deliberate practice. If you want humans to retain the skill to evaluate the agent's output independently, they must regularly exercise that skill in conditions where the agent's output is not available to anchor their judgment.

Operational practices:

The core insight: Scenario 6 required no attacker. The junior analyst lost the ability to catch what the agent missed through the ordinary use of a reliable tool over two years. MIT09 is the only mitigation that directly addresses this class of harm.

💼 Business takeaway

Ask your team: could the people who review your AI's recommendations still do those reviews competently if the AI were turned off tomorrow? Ask whether your review process maintains the human expertise to catch what the agent misses — or whether that skill has quietly eroded.

← Back Defense model →