We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A SaaS company migrated to and started shipping every log to Datadog. For the first few months, the bill was manageable. Then the platform grew. More services, more traffic, more verbose logging. One morning, the finance team flagged a Datadog bill of $87,000 for a single month. The engineering team's monitoring budget for the entire year was $120,000.
The problem wasn't Datadog's pricing. The problem was that nobody was managing the logs. No retention policies. No volume limits. No tiered storage. Every DEBUG message from every service was being indexed, stored, and billed at the same rate as critical ERROR logs.
Log management is the discipline of controlling what you keep, how long you keep it, who can access it, and what it costs. Without it, logging infrastructure either bankrupts you or becomes so unreliable that nobody trusts it.
1. Retention policies. How long do you keep logs at each tier?
| Tier | Retention | Storage Type | Search Speed | Use Case |
|---|---|---|---|---|
| Hot | 3-7 days | SSD/ | Sub-second | Active debugging |
| Warm | 7-30 days | HDD/cheaper index | Seconds | Recent investigation |
| Cold | 30-365 days | Object storage (S3) | Minutes | Compliance, audits |
| Archive | 1-7 years | Glacier/deep archive | Hours | Legal/regulatory |
2. Volume control. Limit what gets collected at the source. Drop DEBUG in production. Sample high-volume, low-value logs. Set per-service quotas.
3. Access control. Not everyone should see every log. Payment logs may contain PII. Authentication logs may contain IP addresses subject to privacy laws. Role-based access control on your logging platform matters.
4. Lifecycle automation. Logs should automatically move through tiers and eventually get deleted. No human should have to manually prune old logs. Set policies once, enforce them automatically.