We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
On August 1, 2012, Knight Capital Group deployed new trading software. Within 45 minutes, a bug caused the system to execute millions of erroneous trades. By the time they pulled the plug, the firm had lost $440 million. They were bankrupt within days.
The postmortem revealed something painful: they had no real-time downtime tracking or deployment monitoring. Nobody noticed the anomaly for 45 minutes because there was no system correlating "we just deployed" with "our trading volume just spiked 10,000%."
Downtime tracking isn't just about knowing your service is down. It's about measuring how long, how often, what caused it, and what the impact was. This data feeds into your SLAs, shapes your engineering priorities, and tells you whether you're getting better or worse at reliability over time.
Every downtime event has a lifecycle that looks like this:
| Phase | Starts When | Ends When | Key Metric |
|---|---|---|---|
| Detection | Service starts failing | Team is aware of the problem | MTTD (Mean Time to Detect) |
| Response | Team is notified | Someone starts investigating | MTTA (Mean Time to Acknowledge) |
| Mitigation | Investigation begins | Impact to users is reduced or eliminated | MTTM (Mean Time to Mitigate) |
| Resolution | Mitigation is in place | Root cause is fully fixed and service is stable | MTTR (Mean Time to Resolve) |
The total downtime, from first failure to full resolution, is what shows up on your status page and counts against your uptime SLA. But the breakdown into phases is where the learning happens. If your MTTD is 20 minutes, investing in better monitoring beats investing in faster fixes.
Most organizations track downtime in an incident management tool. PagerDuty, Opsgenie, Incident.io, or even a shared spreadsheet. Every incident gets logged with start time, end time, severity, affected services, root cause, and a link to the postmortem.