We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Your ran on one server. When something broke, you SSH'd in and ran grep ERROR /var/log/app.log. Simple. Effective. Maybe even a little fun.
Then you moved to . Now you have 47 services running across 200 containers. The user's checkout request touched 8 different services. The error could be in any of them. Are you going to SSH into 200 containers and grep each one?
Log aggregation solves this by shipping logs from every service, every container, every server into a single centralized system where you can search, filter, and correlate them. Instead of hunting through individual machines, you type a query into a search bar and see every log line from every service, sorted by time, filterable by service name, severity, user ID, whatever you need.
Every company past a certain scale treats log aggregation as critical infrastructure. Not optional. Not "nice to have." It's the thing that makes debugging possible when your system spans more than a handful of machines.
Every log aggregation system follows the same basic pattern:
1. Collection. Agents running on each server or container capture log output. These might be Fluentd, Filebeat, Vector, or the Datadog Agent. They watch log files, capture stdout/stderr, and forward everything.
2. Transport. Logs are shipped over the network to a central system. This often goes through a buffer like to handle traffic spikes without losing data.
3. Processing. Logs get parsed, enriched, and transformed. Unstructured text gets parsed into fields. Timestamps get normalized. Metadata gets added (service name, environment, region).
4. Storage. Processed logs land in a searchable store. is the classic choice. Alternatives include Loki (Grafana's log system), ClickHouse, or managed services like Datadog and Splunk.
5. Query/Search. Engineers search and visualize logs through a UI. Kibana for Elasticsearch, for Loki, or the vendor's own interface for managed services.
The entire pipeline needs to handle massive volume. A medium-sized production system can generate gigabytes of logs per hour. A large one generates terabytes per day.