We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
An e-commerce team noticed that API calls between their order service and inventory service occasionally took 5 seconds instead of the usual 50 milliseconds. The code hadn't changed. The database was fast. Both services had low CPU usage.
After days of investigation, they discovered the problem was in the network between two availability zones. A misconfigured security group was causing retransmissions, packets were being dropped and retransmitted with exponential backoff, adding exactly 5 seconds of delay (the default TCP retransmission timeout on Linux).
The application looked healthy. The infrastructure looked healthy. The network was silently eating their performance. Without network monitoring, this problem would have persisted indefinitely. You can't fix what you can't see.
utilization. How much of your available bandwidth is being consumed. If your instance has a 10 Gbps network interface and you're consistently pushing 8 Gbps, you're close to saturation. Traffic spikes beyond capacity cause packet drops.
between services. The round-trip time for packets between two points. Intra-AZ latency is typically under 1ms. Cross-AZ is 1-5ms. Cross-region is 50-200ms. If your intra-AZ latency suddenly jumps to 50ms, something is wrong.
Packet loss. The percentage of packets that never arrive. Even 0.1% packet loss causes significant performance degradation for because every lost packet triggers a retransmission and backoff. Above 1%, user-facing services become noticeably slow.
Connection tracking. How many active TCP connections exist between services. Connection exhaustion (running out of available ports or hitting connection limits) causes connection refused errors.
resolution time. How long it takes to resolve service names to IP addresses. Slow or failing DNS can cripple service-to-service communication.
| Metric | Healthy | Warning | Critical |
|---|---|---|---|
| Inter-service latency | < 1ms (same AZ) | > 5ms | > 50ms |
| Packet loss | 0% | > 0.01% | > 0.1% |
| Bandwidth utilization | < 50% | > 70% | > 90% |
| DNS resolution | < 5ms | > 50ms | > 500ms |