We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.
In short
Logging is the practice of recording discrete events from a running system, each with a timestamp, a severity level, and contextual data, so engineers can later reconstruct what happened. Structured logs written as JSON are searchable by field, while plaintext logs are not, and in production logs are shipped to a central system rather than read off individual machines.
In short
Logging is the practice of recording discrete events from a running system, each with a timestamp, a severity level, and contextual data, so engineers can later reconstruct what happened. Structured logs written as JSON are searchable by field, while plaintext logs are not, and in production logs are shipped to a central system rather than read off individual machines.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Logging across a run of lessons, not one page. These 4 alone are about 66 minutes of step-by-step reading, every one with a quiz.
One place for all logs, the operational backbone of distributed systems.
Write changes to a sequential log before applying them, the foundation of database crash recovery.
Record who did what, when, and why, the immutable trail that compliance, security, and debugging all depend on.
Logs as data, not text, making every log line queryable and machine-parseable.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Centralized Loggingpay once,
yours for life
A log is a record of one thing that happened at one moment in time. A user signed in. A payment failed. A request took 4.2 seconds. Each log line carries at minimum a timestamp, a severity level (DEBUG, INFO, WARN, ERROR), a message, and usually some context such as a user ID, request ID, or service name.
Logs are one of the three pillars of observability, alongside metrics and traces. Metrics answer how much and how often as aggregated numbers. Logs answer what happened in this specific event with the full detail attached. When an alert fires off a metric, the logs are what you read to find the cause.
The difference between a useful log and a useless one usually comes down to structure. A line like User login failed is plaintext and can only be searched with grep. The same event written as JSON with fields like event login_failed, user_id 8842, reason bad_password, ip 10.1.4.9 can be queried by any field, counted, grouped, and filtered. Structured logging is the default for any system you expect to operate at scale.
Application code calls a logging library such as Logback or SLF4J in Java, Winston or Pino in Node.js, or the standard logging module in Python. The library formats the event, attaches the severity and timestamp, and writes it to a destination. In containers the destination is almost always stdout, which the platform captures.
A log shipper or agent then collects those lines and forwards them. Fluent Bit, Filebeat, Vector, and the OpenTelemetry Collector are common agents. They tail files or read the container runtime stream, parse and enrich the events, batch them, and send them over the network to a central store. This decouples the app from the storage so a slow backend never blocks the request path.
The central system indexes the logs for search and retention. Elasticsearch with Kibana, Loki with Grafana, Splunk, and managed services like Datadog or CloudWatch Logs all do this. Because volume is enormous, teams set log levels per service, sample high frequency events, and apply retention windows such as 7 days hot and 90 days in cold storage to control cost. A single busy service can produce gigabytes per hour.
Use logging when you need the full detail of individual events for debugging, auditing, security forensics, or compliance. Logs are the right tool for answering why did this specific request fail. They are the wrong tool for answering what is my p99 latency right now, which is a job for metrics, because computing aggregates over raw logs at query time is slow and expensive.
The main trade-offs are cost and noise. Logging everything at DEBUG in production generates huge volumes that drive up storage and ingestion bills and bury the important lines. Logging too little leaves you blind during an incident. The practical answer is structured logs at INFO by default, ERROR always captured, DEBUG enabled only when investigating, plus a correlation ID on every line so you can stitch one request across many services.
Never log secrets. Passwords, API keys, full credit card numbers, and personal data leaking into logs is one of the most common security and GDPR failures, because logs are often readable by far more people than the database. Scrub or hash sensitive fields before they are written.
Picture an e-commerce checkout that spans three services: an API gateway, a payments service, and an inventory service. A customer reports that their order vanished. With unstructured logs you would SSH into three machines and grep through plaintext, hoping the timestamps line up.
With structured logging done properly, the gateway generates a request ID such as req-7f3a and passes it in a header. Every service logs that same request_id field on each line. In Kibana or Grafana you search request_id:req-7f3a and instantly see the full ordered story: gateway accepted the request, payments charged the card, inventory threw an ERROR because stock hit zero between the check and the reserve, and the order was rolled back.
That single query, made possible by structured fields and a shared correlation ID, turns a multi hour cross team investigation into a one minute lookup. This is why logging is treated as core infrastructure and not an afterthought.
Elastic (ELK Stack)
Elasticsearch, Logstash, and Kibana are the most widely deployed open source log search and visualization stack.
Grafana Loki
Indexes only log labels instead of full text, which makes it cheaper to run at high volume alongside Grafana dashboards.
Splunk
Enterprise platform used heavily for log analytics, security monitoring, and compliance auditing.
Amazon CloudWatch Logs
AWS managed service that collects logs from EC2, Lambda, and containers for search, alarms, and retention.
Logging records individual events with full detail so you can reconstruct exactly what happened. Monitoring watches aggregated metrics over time and alerts when something crosses a threshold. You monitor to know that something is wrong, then read logs to find out why.
Structured logs write each event as fields, usually JSON, so you can search, filter, count, and group by any field such as user_id or status_code. Plaintext can only be searched with text matching like grep, which makes querying at scale slow and unreliable.
Use DEBUG for detailed diagnostic output during development, INFO for normal events worth recording, WARN for unexpected but recoverable situations, and ERROR for failures that need attention. In production, run at INFO by default and enable DEBUG only while investigating an issue.
In a distributed system a single request touches many machines, and containers are short lived so their local logs disappear when they restart. Shipping logs to a central store lets you search across all services at once, correlate events with a request ID, and keep history for auditing.
Passwords, API keys, session tokens, full credit card numbers, and personal data. Logs are usually readable by far more people than your database, so leaking secrets into them is a common security and privacy breach. Scrub, mask, or hash sensitive fields before writing.
Write-Ahead Logging
Write changes to a sequential log before applying them, the foundation of database crash recovery
Audit Logging
Record who did what, when, and why, the immutable trail that compliance, security, and debugging all depend on
Structured Logging
Logs as data, not text, making every log line queryable and machine-parseable
Security Audit Logging
Recording security-relevant events with tamper-proof, queryable audit trails for compliance and incident investigation
Log Levels
The severity hierarchy that determines which messages matter and which are noise
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Elasticsearch
A distributed search and analytics engine built on Apache Lucene. Powers full-text search, log analysis, and real-time analytics at scale.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
This page explains the idea. The "Centralized Logging" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Centralized Logging lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Logging in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
