What it does: Actions are designed so that executing them multiple times is safe — the outcome is the same whether the action ran once or ten times. A unique idempotency key per request allows the receiving system to detect and ignore duplicate requests.

Why agent pipelines need this: Retries, circuit breaker recovery, and thundering herd patterns all cause duplicate requests. Without idempotency, a recovered pipeline sends duplicate emails, double-charges customers, or creates duplicate database records. With idempotency keys, the second and third execution of the same logical action are recognized as duplicates and discarded.

Implementation: Generate a unique UUID per logical operation at the orchestrator level. Pass it through every agent as a request ID. At the terminal action agent, check the idempotency key against a seen-keys store before executing. If the key has been seen, return the cached result without executing again.

Directly fixes Scenario 5's duplicate email problem: When the customer service pipeline recovered and all 6 agents resumed, without idempotency they sent duplicate responses to every ticket. With idempotency keys, each ticket's response could only be sent once — the second attempt would be recognized as a duplicate and discarded.

What it monitors: Per-agent output volume (deviation from baseline), output quality metrics (validation pass rate, field completeness scores), latency (P99 response time), error rates, and downstream action volume (emails sent per hour, transactions per hour).

Key principle: The monitoring system must be independent of the shared dependencies it monitors. A monitoring agent that calls the same LLM API as the pipeline it watches will be unavailable during exactly the outage it's supposed to detect — as occurred in Scenario 5.

What it does: In a controlled environment, deliberate failures are injected into the pipeline: an agent is made to return truncated data, a dependency is made temporarily unavailable, a circuit breaker is tripped intentionally. The test verifies that the fallback behaviors, circuit breakers, and isolation mechanisms engage as designed — and measures the actual blast radius of each failure mode before it occurs in production.

Why testing is non-negotiable: Resilience mechanisms that have never been tested under realistic load often fail in ways their designers didn't anticipate. The circuit breaker configuration may be too sensitive (trips on normal load variation) or too lenient (never trips on real failures). Chaos testing is the only way to know.