We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
An API team reported "average response time: 50ms." The leadership team was thrilled. Then the support team mentioned they were getting 200 complaints per day about slow responses.
When they looked at the histogram, the distribution of all response times, not just the average, they found a bimodal distribution. 95% of requests completed in 20ms (fast cache hit). 5% of requests took 2-4 seconds (cache miss + cold database query). The average of 50ms was technically correct but completely hid the 5% of users having a terrible experience.
Histograms show you the shape of your data. An average collapses all that shape into a single number. When you're trying to understand system performance, the shape matters enormously, and histograms are the metric type that preserves it.
A histogram metric divides a range of values into predefined buckets and counts how many observations fall into each bucket.
For a response time histogram with buckets at 10ms, 50ms, 100ms, 250ms, 500ms, 1s, and 5s:
| Bucket (le) | Count | Meaning |
|---|---|---|
| 10ms | 4,200 | 4,200 requests completed in <= 10ms |
| 50ms | 8,900 | 8,900 requests completed in <= 50ms |
| 100ms | 9,400 | 9,400 requests completed in <= 100ms |
| 250ms | 9,700 | 9,700 requests completed in <= 250ms |
| 500ms | 9,850 | 9,850 requests completed in <= 500ms |
| 1s | 9,950 | 9,950 requests completed in <= 1s |
| 5s | 10,000 | All 10,000 requests completed in <= 5s |
| +Inf | 10,000 | Total count (always matches) |
Buckets are cumulative, each includes all smaller buckets. From this data, you can calculate percentiles: the P95 is somewhere between 250ms and 500ms (9,500th request out of 10,000). The P99 is between 500ms and 1s.
In , a histogram metric actually creates three time series per bucket:
http_request_duration_seconds_bucket{le="0.05"}, count of requests <= 50mshttp_request_duration_seconds_sum, total sum of all observed durationshttp_request_duration_seconds_count, total number of observationsChoosing bucket boundaries is the key decision. If your SLO is "95% of requests under 300ms," you need a bucket at 300ms. If all your buckets are at 1s, 5s, 10s, you can't measure anything useful below 1 second.