Root Cause Analysis

A checkout is returning 500s. The investigation finds that the payment service ran out of database connections. The fix is obvious: bump the connection pool from 20 to 100. Ship it. Done.

Except the same incident recurs the following week.

Etsy's engineering blog has a well known 2016 post about exactly this pattern. The team stopped at the first convenient answer, shipped the pool increase, and the underlying slow query without an index caused pool exhaustion again as soon as traffic grew. The real fix was the index, not the pool size. But even the index was not the root cause. The schema change that introduced the slow query had bypassed review because the author had approver rights on their own PR.

Root cause analysis is the discipline of pushing past the first convenient answer. The tools are simple: Five Whys, fishbone diagrams, structured postmortem templates. The hard part is organizational willpower to keep asking why when the third answer already feels good enough.

This lesson covers the Five Whys technique from Toyota, the Ishikawa fishbone for multi-cause incidents, and the specific failure patterns that make teams stop their analysis too early.