We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A payment processing company had logs scattered across 30 different servers. When a customer disputed a charge, the support team had to file a ticket to engineering. An engineer would then SSH into multiple servers, searching for the transaction ID across different log files with different formats, in different time zones, with different retention policies.
What should have been a 4-minute lookup took 4 hours on average. Multiply that by 50 disputes per week and you've got an engineer spending half their time just finding log entries.
They switched to centralized logging, a single cluster where every service shipped its logs. Now, the support team typed the transaction ID into Kibana and saw the entire lifecycle of that transaction across every service in seconds. No SSH. No engineering tickets. No 4-hour hunts.
Centralized logging means every log from every service, every server, and every environment flows into a single searchable system. It's the difference between a library with a card catalog and a warehouse of unsorted papers.
A centralized logging system has three core layers:
Collection layer. Lightweight agents on every machine or container capture log output and forward it. Filebeat reads log files. Fluentd collects from multiple sources. In , DaemonSets ensure every node has a log collector.
Processing layer. An ingestion pipeline that parses, transforms, and enriches logs before storage. This is where unstructured text like [2026-04-04 10:30:12] ERROR PaymentService: Timeout connecting to Stripe gets parsed into structured fields: timestamp, level, service, message. Logstash, Fluentd, and Vector all serve this role.
Storage and query layer. The searchable backend. is the industry standard, it indexes every field for fast full-text search. Alternatives: Grafana Loki (stores logs cheaply, indexes only labels), ClickHouse (columnar, great for analytics), and managed services like Datadog, Splunk, and Sumo Logic.
| Component | Open Source Options | Managed Options |
|---|---|---|
| Collection | Filebeat, Fluentd, Vector | Datadog Agent, Splunk Forwarder |
| Processing | Logstash, Fluentd, Vector | Built into managed platforms |
| Storage | Elasticsearch, Loki, ClickHouse | Datadog, Splunk Cloud, AWS CloudWatch |
| Visualization | Kibana, | Vendor dashboards |
The ELK stack (Elasticsearch + Logstash + Kibana) dominated for years. Now many teams use the "EFK" stack (Elasticsearch + Fluentd + Kibana) or Grafana Loki for cost-conscious setups.