A research pipeline has 9 agents: 1 orchestrator, 4 retrieval agents (different data sources), 3 synthesis agents, 1 output formatter. All 4 retrieval agents share a vector database. The vector database experienced a 12-second maintenance window at 11:03 AM.

All 4 retrieval agents received timeouts simultaneously. Each was configured to retry 3 times with a 2-second delay — no exponential backoff, no jitter. At 11:03:12, all 4 agents sent retry requests simultaneously. The vector database, now back online, was hit with 4× normal load during its recovery. It timed out again. All 4 agents retried again. The thundering herd pattern repeated. By 11:07, all 9 agents had stopped processing. The vector database was healthy; what was unhealthy was the retry pattern overwhelming its recovery.

Fix: Exponential backoff with jitter (each agent waits a random interval before retrying, increasing with each attempt). Circuit breaker on each retrieval agent (stop retrying after 3 consecutive failures; only attempt again after a recovery window). Bulkhead isolation (database connection pools per agent, so one agent's retry storm can't consume connections belonging to another agent).

A compliance pipeline uses a document analysis agent to extract structured fields from contracts. One document caused the LLM to hallucinate a field: "exclusivity_period": "unlimited" — a field the contract did not contain. The extraction agent returned this with full confidence alongside real extracted fields.

Three downstream agents acted on it: a risk scoring agent treated "unlimited exclusivity" as extreme risk, elevating the contract to manual review. A summary agent included the non-existent clause in its summary, which was sent to the client. A comparison agent flagged it as a material deviation from the standard template — triggering an escalation email to legal. All three agents processed a hallucination as fact because there was no validation layer between extraction and reasoning.

What was missing: Output validation that checked extracted fields against the known contract schema (fields that don't exist in the template should be flagged for human review). A confidence threshold below which extraction outputs are routed to human review before downstream processing. Idempotent downstream actions (escalation emails should require human confirmation before sending).