Bottleneck Identification

A checkout service was slow. P99 latency had crept from 200 ms to 1.2 seconds over two months. The team looked at the database dashboard, saw query times rising, and tripled the Postgres instance from db.r6g.xlarge to db.r6g.4xlarge. Cost went from $580/month to $1,740/month.

Latency did not move.

The database was never the bottleneck. A flame graph captured the next week showed that 48 percent of CPU time on the application servers was spent in a regex compilation inside a middleware layer that ran on every request. The regex was recompiled from source on every call instead of being compiled once at startup. The fix was a single line change: move the compile to init. P99 dropped to 180 ms. The team downgraded the database back to xlarge.

This is the most common failure mode in performance work: guessing which component is the bottleneck instead of measuring. Brendan Gregg at Netflix published the USE method in 2012 specifically to replace intuition with a checklist, because experienced engineers skip to their favorite metric and miss the true bottleneck in about 40 percent of cases.