How do you design a distributed logging system?

Estimate the volume first, then build five stages: an agent on every machine that reads logs, adds labels and buffers to disk; Kafka as a durable buffer so a slow store never reaches the apps; processors that parse, mask secrets and sample; a hot searchable store for recent days plus a cheap object-storage archive for months; and a query UI with alerts. Choose full-text, label or column indexing based on how people search, and plan how the pipeline behaves in an error storm.

Why use Kafka in a logging pipeline?

Kafka keeps hours of logs on disk between the agents and the stores. If a store is slow or down, consumers fall behind and catch up later, and the agents and apps never notice. It also allows replaying logs after a processing bug and lets several systems read the same logs.

What is the difference between Elasticsearch and Loki for logs?

Elasticsearch, and OpenSearch, build a full-text index of every word, so any word is found fast, at a high write and storage cost. Grafana Loki indexes only labels, such as service and environment, and stores the text in compressed chunks in object storage, so it is much cheaper to run, but searches that labels cannot narrow must scan more data.

How much storage does a logging system need?

Work it out from lines per second x bytes per line. For example, 500,000 lines a second at 500 bytes is 250 MB a second, or 21.6 TB a day raw. Assuming 5x compression, 7 days hot with a replica is about 60.5 TB, and a year in object storage is about 1.58 PB. Measure your own compression ratio, because it varies a lot.

How do you stop logging from slowing down the application?

Apps write to stdout or a local file and never wait on the network. An agent ships the logs in the background with its own CPU and memory limits and a disk buffer. Kafka absorbs slow stores. In a burst, sample DEBUG lines and drop the least useful lines first, never ERROR, and count everything you drop.

What is the difference between a logging framework and a logging system?

A logging framework is the library inside one program that formats and writes log lines, which is a low-level design question. A distributed logging system is the platform that collects those lines from thousands of machines, stores and indexes them, and makes them searchable, which is a high-level design question.

What are hot and cold tiers in log storage?

The hot tier keeps recent logs, often 7 to 30 days, on fast disks with an index so searches take seconds. The cold tier keeps older logs as compressed files in object storage such as S3, which is far cheaper, and loads them back only when needed.

Should you sample logs?

Sample only what is cheap to lose. Keep every ERROR and WARN line, sample DEBUG lines (for example 1 in 10), and collapse repeated identical lines into one with a count. If DEBUG is 60% of the volume, keeping 1 in 10 leaves 46% of the bytes. Count and report whatever you drop.

System design interview guide

Design a Distributed Logging System: Agents, Kafka, Indexing, Retention and Search

Short answer

A distributed logging system collects log lines from every server, keeps them safe and makes them searchable. Apps write structured JSON logs to stdout. An agent on each machine (Fluent Bit, Vector or the OpenTelemetry Collector) adds labels such as service and pod, batches the lines and buffers to disk. Kafka sits in the middle as a durable buffer, so a slow store never pushes back on the apps. Processors parse lines, mask secrets and personal data, and sample noisy DEBUG lines. Recent logs go to a hot store: a full-text index (OpenSearch), a label index with compressed chunks (Grafana Loki) or a column store (ClickHouse). Older logs go to cheap object storage such as S3. At 500,000 lines a second of 500 bytes, that is 21.6 TB a day raw, so indexing and retention choices set the cost.

in this guide

Design a distributed logging system, part by part

8 parts open to everyone · 3 with a free account · 3 with the course

Requirementsopen
Scale estimatesopen

Why this question is asked

It tests whether a candidate can design a write-heavy pipeline where the data is huge, mostly never read, and urgently needed at the worst moment. A weak answer draws 'apps send logs to Elasticsearch' and stops. A strong answer estimates the volume and turns it into storage cost. It separates collection from storage with a buffer, so the logging system can never take down the apps. It chooses how to index with a reason, because indexing every word of 21.6 TB a day is the single biggest cost. It plans retention tiers, handles bursts by sampling and dropping in a known order, and treats logs as sensitive data that can leak passwords and personal details.

This page is the free part.

The course goes deeper on the distributed logging system design

₹499 in India$49 everywhere elseonce, for the whole course

The System Design course covers the distributed logging system design across a run of lessons, not one page. These 4 alone are about 56 minutes of step-by-step reading, every one with a quiz.

Live from the course bench

Before you design Distributed Logging System, try the questions it rests on.

Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.

Two engineers sketching a system design on a whiteboard

where the work happens

Stuck on a card?

Ask the course tutor. It answers with a real diagram, drawn for your question.

Ask the tutor

Laying out the cards…

Requirements

Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.

Functional requirements

Collect logs from every server, container and service, with labels: service, environment, host or pod, region
Accept structured (JSON) logs and plain text, and keep a trace ID (an ID that follows one request across all services) so logs can be joined with traces

Go deeper

This guide is built on these 6 lessons. Each one teaches one piece of the distributed logging system design above, step by step.

Practice with 770 system design lessons

Lifetime access for ₹499 in India or $49 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.

Get lifetime access See all interview problems

Two courses by the author of this page:

System Design

770 lessons · 18 free to read

₹499 in India · $49 elsewhere, once

Get System Design

You own this course

AI Engineering

210 lessons · 10 free to read

₹999 in India · $49 elsewhere, once

Get AI Engineering

You own this course

Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor

Save your progress and get the interview checklist PDF

System Design Interview Cheat Sheet: the 7 interview stages, the 10 points a scorecard grades, 12 classic systems. 18 pages. Free with an account.

Every program writes a diary as it runs: 'order 812 paid', 'database timeout after 2 s', 'user 42 logged in'. Each line is a log. On one server you read the file. Now run 5,000 servers and hundreds of services (separate programs that each do one job). A user says checkout failed at 10:42. The answer is in one line, on one of 5,000 machines, among 500,000 new lines a second. Machines are replaced every day, so their local files vanish. A logging system collects every line from every machine into one place, keeps it safe, and lets an engineer search all of it in seconds. It must never slow down the apps it watches, and it must survive the worst moment, when everything is failing and every server suddenly logs ten times more.

A distributed logging system collects log lines from many servers, stores them centrally, and makes them searchable. First, some terms. A log line is one record a program writes, such as an error with a time and a message. Structured logs are written as fields (JSON is common), for example level=ERROR, service=checkout, trace_id=abc, instead of free text, so machines can filter them. An agent is a small program on each server, such as Fluent Bit, Vector or the OpenTelemetry Collector, that reads the logs and ships them. Kafka is a system that stores a stream of messages on disk, in order, so other programs can read them later. Object storage, such as Amazon S3, is cheap storage for whole files. The usual design has five stages. Collect: apps write to standard output or a file, and an agent on each machine reads it. It adds labels such as the service and the pod (one running copy of an app in Kubernetes), and buffers on disk if the next stage is slow. Buffer: a durable queue, usually Kafka, takes everything, so a slow or broken store never pushes back on the apps. Process: consumers (programs that read from Kafka) parse lines, remove secrets and personal data, drop or sample noise, and route by team. Store: a hot tier keeps recent logs ready to search. It is one of three kinds: a full-text index of every word (Elasticsearch or OpenSearch), a small index of labels with the text in compressed chunks (Grafana Loki), or a column store (ClickHouse). Then a cheap cold tier in object storage keeps logs for months or years. Query: a UI such as Grafana or Kibana, plus alerts on error patterns. With 5,000 servers writing 100 lines a second of 500 bytes each, that is 250 MB a second, 21.6 TB a day before compression, which is why indexing choices and retention decide the cost.

Roni Das

21 October 2026 · 29 min read

Where it shows up

Asked as 'design a logging system', 'design a distributed logging system', 'design a centralised log aggregation system' or 'design Splunk or the ELK stack' (Elasticsearch, Logstash and Kibana). Common at cloud, infrastructure, observability and SRE teams, and at any company with many microservices (many small services). It is also a frequent follow-up in other designs: 'how would you debug this in production?' leads straight here.

Spec sheeta distributed logging system

01Ingest rate: 500,000 lines a second on average, 1.5 million at peak
02Bytes per second: 250 MB a second on average, 750 MB at peak
03Raw volume per day: 21.6 TB a day
04Storage: hot and cold tiers: about 60.5 TB hot, about 1.58 PB cold

worked through below, with the maths

Data modelfree account

Deep dives (8)free account

The hard parts of this design are taught step by step in the System Design course: Log Aggregation, Centralized Logging and Structured Logging.

Log Aggregation
Collecting logs from every service into one searchable place, finding the needle in the haystack.
14 min · diagrams · quiz
Centralized Logging
One place for all logs, the operational backbone of distributed systems.
15 min · diagrams · quiz
Structured Logging
Logs as data, not text, making every log line queryable and machine-parseable.
14 min · diagrams · quiz
Log Management
Retention, rotation, access control, and lifecycle policies, keeping logs useful without drowning in them.
13 min · diagrams · quiz

all part of

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499in India, by UPI

$49everywhere else, by PayPal

Get the System Design course Read 18 lessons free first

You own this course

Continue with Log Aggregation

pay once,
yours for life

Search recent logs by label, time range and free text, with results in seconds

Keep logs for a set time per team or log type: for example 7 days searchable, 1 year archived

Alert on patterns, such as a spike in ERROR lines for one service

Remove secrets and personal data before storage, and control who can read which logs

Non-functional requirements

Never slow down or crash the apps: logging must be asynchronous, meaning the app never waits for the log store
Durable under bursts: survive an error storm of 10 times normal volume, and a storage outage of a couple of hours, without losing ERROR lines
Fresh: a line is searchable within about a minute of being written
Cheap: storage cost grows with volume, so most bytes must sit in cheap, compressed storage
Multi-tenant: many teams (tenants) share it, and one noisy service must not crowd out everyone else
Scale out by adding machines at every stage

Back-of-envelope scale estimates

Show your math. Pulling numbers from thin air signals you have not thought about the load.

Ingest rate

500,000 lines a second on average, 1.5 million at peak

Interview assumptions: 5,000 servers, each writing 100 log lines a second on average. 5,000 x 100 = 500,000 lines a second. Plan for a peak of 3 times that, 1,500,000 lines a second, because error storms are exactly when logs matter most.

Bytes per second

250 MB a second on average, 750 MB at peak

Assume an average structured (JSON) line of 500 bytes. 500,000 x 500 = 250,000,000 bytes, which is 250 MB a second. At peak, 1,500,000 x 500 = 750 MB a second.

Raw volume per day

21.6 TB a day

250 MB a second x 86,400 seconds in a day = 21.6 TB a day before compression. This number drives every cost decision in the design.

Storage: hot and cold tiers

about 60.5 TB hot, about 1.58 PB cold

Assume logs compress 5 times; real ratios depend heavily on the data, so measure your own. 21.6 / 5 = 4.32 TB a day. Hot tier: 7 days kept searchable with one replica (one extra copy), 4.32 x 7 x 2 = 60.5 TB on fast disks. Cold tier: a year in object storage, 4.32 x 365 = 1,577 TB, about 1.58 PB (a petabyte is 1,000 TB). This ignores the index; a full-text index adds a large share on top of the hot tier, which is why the indexing choice matters.

Kafka buffer for a storage outage

1.8 TB of logs, 5.4 TB of disk with 3 copies

Goal: if the store is down for 2 hours, keep everything. 250 MB a second x 7,200 seconds = 1.8 TB. With a replication factor of 3, a common choice (Kafka keeps 3 copies of each partition), that is 5.4 TB of disk across the Kafka cluster, before compression. This is at the average rate. If the outage happens during a 3 times peak, you need three times as much.

Kafka partitions

at least 75

A topic is a named stream of messages in Kafka. A partition is one ordered slice of a topic that only grows at the end. In each consumer group (one team of readers), only one consumer reads it at a time. Planning assumption: each partition carries at most 10 MB a second. Peak 750 MB a second / 10 = 75 partitions. Round up for growth, for example to 96 or 128, because adding partitions later reshuffles which keys go where.

Agent disk buffer

a 1 GB buffer holds about 5.6 hours on one server

One server writes 100 x 500 = 50 KB a second. 1 GB / 50 KB a second = 20,000 seconds, about 5.6 hours. So if Kafka is unreachable, each agent can keep logs on its own disk for hours before it must drop anything, at normal volume. In a 10 times burst, that falls to about 33 minutes.

Sampling saves

keeping 1 in 10 DEBUG lines leaves 46% of the bytes

Assume DEBUG lines are 60% of the volume. Keep every ERROR and WARN, keep 1 in 10 DEBUG lines: 1 - 0.6 x 0.9 = 0.46, so 46% of the bytes remain and storage cost roughly halves, without losing a single error.

High-level architecture

Draw five stages, left to right. One: collection. Apps write logs to standard output and standard error. In Kubernetes (a system that runs containers across many machines), the container runtime writes those streams to files on the node, the machine. An agent runs once on every node (a DaemonSet). The agent, for example Fluent Bit, tails the files, which means it reads new lines as they are added. It adds labels from Kubernetes (namespace, pod, container), batches lines, compresses them, and sends them on. If the next stage is slow, it buffers on its own disk. Two: the buffer, a Kafka cluster. Agents write to log topics, split by team or log type, partitioned so that load spreads evenly. Kafka keeps a few hours of logs on disk, so the stores behind it can fall behind, restart or be upgraded without any effect on the apps. Three: processing. A pool of consumers, which can be Vector, Logstash, Fluentd or your own code, reads from Kafka and parses each line into fields, drops or masks secrets and personal data, samples noisy DEBUG lines, enforces per-service quotas, and routes each line to the right store and retention. Four: storage, in two tiers. The hot tier holds recent days and answers searches in seconds. It can be OpenSearch or Elasticsearch (a full-text index), Grafana Loki (a label index plus compressed chunks in object storage), or ClickHouse (a column store, which Uber moved to). The cold tier is compressed files in object storage such as S3, kept for months or years for audits and rare investigations, and loaded back only when needed. Five: query and alerts. Grafana or Kibana for search and dashboards, and alert rules that run on the stream or on the hot store, for example 'more than 100 ERROR lines a minute from checkout'. Every line carries a trace ID where possible, so an engineer can jump from a log line to the full trace of that request.

In a real interview, sketch this on the whiteboard before diving into any single box.

One error line, from the app to an engineer's screen

Follow one ERROR from the checkout service. This is a good order to explain it in an interview.

1
The app writes it
checkout writes one JSON line to standard output: time, level=ERROR, message 'payment timeout', order_id, trace_id. Writing to stdout is fast and never waits on the network.
2
The agent picks it up
Fluent Bit on the node reads the new line from the container's log file within a second, adds labels (service=checkout, pod, namespace, region) and puts it in a batch.
3
Batch to Kafka
Every second or so, or when the batch is full, the agent compresses the batch and sends it to the Kafka topic for the payments team. If Kafka does not answer, the batch waits in the agent's disk buffer.
4
Process
A consumer reads the batch, checks the JSON, masks anything that looks like a card number or password, keeps the ERROR line (errors are never sampled), and writes it to the hot store and to the archive stream.
5
Index
The hot store adds the line to its index. In OpenSearch every word becomes searchable; in Loki only the labels are indexed and the text sits in a compressed chunk. Either way, it is searchable within about a minute.
6
Alert
An alert rule counts ERROR lines from checkout. The count crosses its limit, and the engineer on duty gets an alert on their phone, with a link to a saved search.
7
Search and jump
The engineer searches service=checkout, level=ERROR for the last 15 minutes, finds the line, and follows its trace_id to the trace, which shows the payment provider call that timed out.

Core components

Walk through each service. The interviewer wants to hear what each one owns, not just the names.

Log agent (per node)

Fluent Bit, Vector, the OpenTelemetry Collector or Filebeat. Tails files or reads stdout, adds labels, batches, compresses, retries, and buffers to disk. It must use little CPU and memory, because it runs on every machine next to the apps.

Durable buffer (Kafka)

Takes every line and keeps hours of history, so the rest of the pipeline can be slow, down or replayed without affecting the apps. Topics per team or log type let each have its own retention and consumers.

Processors

Stateless consumers that parse, enrich, mask secrets and personal data, sample, apply per-service quotas and route lines. They scale by adding consumers, up to one per partition.

Hot store and index

Recent logs, searchable in seconds. A full-text index (OpenSearch, Elasticsearch), a label index with compressed chunks (Grafana Loki), or a column store (ClickHouse). The choice sets most of the cost.

Cold archive

Compressed files in object storage (S3, GCS, Azure Blob), partitioned by date and service. Cheap to keep for a year or more; slow to search, so data is loaded back only for audits or rare investigations.

Query UI and alerting

Grafana or Kibana for search, saved queries and dashboards. Alert rules on error counts and patterns, with links back to the search and to traces.

the deep dive

Data model

log record (one line). This follows the OpenTelemetry logs data model, which names Timestamp, SeverityText and SeverityNumber, Body, Attributes, TraceId and SpanId. Keeping trace_id is what…

free with an account

Every table for a distributed logging system, its keys, and which store fits it and why.

One tap with Google. Free, no card.

the deep dive

Deep dives

Structured logs and what every line should carry. Free text, such as 'Payment failed for order 812 after 2000ms', is easy to write and hard to use. To find all payment failures over 1 second, a machine has to guess…

free with an account

8 deep dives on the parts that break at scale, with 3 diagrams: Structured logs and what every line should carry; Collecting logs from every machine without hurting the apps; Why Kafka sits in the middle.

One tap with Google. Free, no card.

the deep dive

Trade-offs to discuss

Full-text index vs label index vs column store. A full-text index (OpenSearch) finds any word fast but pays heavily on every write and in disk. A label index (Loki) is cheap to write and store but slow for searches…

free with an account

6 decisions an interviewer expects you to defend, each with the alternative and why it loses here.

One tap with Google. Free, no card.

for course owners

What the interviewer asks next

During an incident, searches are slow and the newest logs are 20 minutes old. What is happening?

with the System Design course

5 follow-up questions with model answers, the interview scorecard for this question, and a mock interview on Design a distributed logging system with the AI tutor.

Unlock with the course · $49One payment, lifetime access to every lesson.

How a distributed logging system actually does it

Uber ran Elasticsearch-based logging from 2014, at 20+ clusters per region, and moved to ClickHouse in a schema-agnostic design, reporting 300,000 logs a second per node and hardware cost cut by more than half (2021). Grafana Loki indexes only labels and keeps log text in compressed chunks in object storage. Fluent Bit documents both memory and filesystem buffering and what happens at each limit. Kubernetes documents the node-level agent pattern, with containers writing to stdout and stderr. The OpenTelemetry logs data model gives a common shape for log records, including trace and span IDs.

Sources

the question most readers have next

What would you have to get right to run the distributed logging system design for real users?

The design on this page is the shape. Running it means handling the failures, the load and the data that the shape does not show. The lessons below teach those parts, starting with "Log Aggregation".

Where this is used in industry

The real examples, each with its source, are on this page: How a distributed logging system actually does it.

The lessons that teach the next part

Collecting logs from every service into one searchable place, finding the needle in the haystack.
Log AggregationSystem Design course14 min
One place for all logs, the operational backbone of distributed systems.
Centralized LoggingSystem Design course15 min
Logs as data, not text, making every log line queryable and machine-parseable.
Structured LoggingSystem Design course14 min
Retention, rotation, access control, and lifecycle policies, keeping logs useful without drowning in them.
Log ManagementSystem Design course13 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free System Design Interview cheat sheet (PDF)

Where this is taught

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499 in India by UPI$49 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Frequently asked questions

System Design · observability monitoring · 13 min

India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.

NEWSystem Design Hands-On

Included in the System Design course: real systems, built live

Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.

Watch the free 15 minutes See System Design Hands-On

Free PDF · 79 pages

25 real tech interview questions, answered in full

Five from each of our AI, GenAI, Inference, Maths for ML and Forward Deployed packs. Each with a diagram and a model answer. Free by email.

Also by the same author

210 lessons · about 219 hours · 10 free to read

₹999 in India by UPI$49 elsewhere by PayPal, once, for life

See the AI Engineering course

what a reader said

It is really nice and good Please join the course

Gyan Kumar, Analyst, Bangalore · System Design Masterclass · read 3 lessons · all reviews

Design a Distributed Logging System: Agents, Kafka, Indexing, Retention and Search

Design a distributed logging system, part by part

Why this question is asked

The course goes deeper on the distributed logging system design

Before you design Distributed Logging System, try the questions it rests on.