We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
When organizations realized they needed centralized log management but couldn't afford Splunk's licensing, they turned to the ELK stack, three open-source projects that together provide similar capabilities at a fraction of the cost.
E. : a distributed search engine that indexes and stores log data L. Logstash: a pipeline that collects, transforms, and ships logs to Elasticsearch K. Kibana: a web UI for searching logs, building dashboards, and creating visualizations
The ELK stack (now often called the Elastic Stack, with Beats added as lightweight data shippers) became the most widely deployed open-source log management solution. Thousands of organizations run it, from startups to Netflix, Wikipedia, and LinkedIn.
The trade-off is clear: ELK costs less in licensing but more in operational effort. You're responsible for Elasticsearch cluster management, capacity planning, index lifecycle policies, and performance tuning. Splunk handles all of that for you but charges accordingly.
is the core, a distributed, full-text search engine built on Apache Lucene. It shards data across nodes for horizontal scaling, replicates shards for availability, and indexes every field for fast search. For logs, it's configured with time-based indices (one index per day) that roll over and get deleted according to retention policies.
Logstash is the processing pipeline. It reads from inputs (files, , Kafka, Beats), applies filters (parse JSON, extract fields from unstructured text, add metadata), and writes to outputs (Elasticsearch, S3, another Kafka topic). It's powerful but resource-heavy, a common replacement is Fluentd or Vector.
Kibana is the UI layer. It provides Discover (free-text log search), Dashboard (visualization panels), and Lens (drag-and-drop chart builder). Kibana also includes alerting, machine learning anomaly detection, and Canvas for presentation-ready visualizations.
Beats are lightweight, single-purpose data shippers: Filebeat (log files), Metricbeat (system metrics), Packetbeat (network data), Auditbeat (audit events). They replaced Logstash as the collection agent on individual hosts due to their lower resource footprint.