Logging
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.
What is Logging?
In short
Logging is the practice of recording discrete events from a running system, each with a timestamp, a severity level, and contextual data, so engineers can later reconstruct what happened. Structured logs written as JSON are searchable by field, while plaintext logs are not, and in production logs are shipped to a central system rather than read off individual machines.
What logging actually is
A log is a record of one thing that happened at one moment in time. A user signed in. A payment failed. A request took 4.2 seconds. Each log line carries at minimum a timestamp, a severity level (DEBUG, INFO, WARN, ERROR), a message, and usually some context such as a user ID, request ID, or service name.
Logs are one of the three pillars of observability, alongside metrics and traces. Metrics answer how much and how often as aggregated numbers. Logs answer what happened in this specific event with the full detail attached. When an alert fires off a metric, the logs are what you read to find the cause.
The difference between a useful log and a useless one usually comes down to structure. A line like User login failed is plaintext and can only be searched with grep. The same event written as JSON with fields like event login_failed, user_id 8842, reason bad_password, ip 10.1.4.9 can be queried by any field, counted, grouped, and filtered. Structured logging is the default for any system you expect to operate at scale.
How logging works under the hood
Application code calls a logging library such as Logback or SLF4J in Java, Winston or Pino in Node.js, or the standard logging module in Python. The library formats the event, attaches the severity and timestamp, and writes it to a destination. In containers the destination is almost always stdout, which the platform captures.
A log shipper or agent then collects those lines and forwards them. Fluent Bit, Filebeat, Vector, and the OpenTelemetry Collector are common agents. They tail files or read the container runtime stream, parse and enrich the events, batch them, and send them over the network to a central store. This decouples the app from the storage so a slow backend never blocks the request path.
The central system indexes the logs for search and retention. Elasticsearch with Kibana, Loki with Grafana, Splunk, and managed services like Datadog or CloudWatch Logs all do this. Because volume is enormous, teams set log levels per service, sample high frequency events, and apply retention windows such as 7 days hot and 90 days in cold storage to control cost. A single busy service can produce gigabytes per hour.
When to use it and the trade-offs
Use logging when you need the full detail of individual events for debugging, auditing, security forensics, or compliance. Logs are the right tool for answering why did this specific request fail. They are the wrong tool for answering what is my p99 latency right now, which is a job for metrics, because computing aggregates over raw logs at query time is slow and expensive.
The main trade-offs are cost and noise. Logging everything at DEBUG in production generates huge volumes that drive up storage and ingestion bills and bury the important lines. Logging too little leaves you blind during an incident. The practical answer is structured logs at INFO by default, ERROR always captured, DEBUG enabled only when investigating, plus a correlation ID on every line so you can stitch one request across many services.
Never log secrets. Passwords, API keys, full credit card numbers, and personal data leaking into logs is one of the most common security and GDPR failures, because logs are often readable by far more people than the database. Scrub or hash sensitive fields before they are written.
A concrete example
Picture an e-commerce checkout that spans three services: an API gateway, a payments service, and an inventory service. A customer reports that their order vanished. With unstructured logs you would SSH into three machines and grep through plaintext, hoping the timestamps line up.
With structured logging done properly, the gateway generates a request ID such as req-7f3a and passes it in a header. Every service logs that same request_id field on each line. In Kibana or Grafana you search request_id:req-7f3a and instantly see the full ordered story: gateway accepted the request, payments charged the card, inventory threw an ERROR because stock hit zero between the check and the reserve, and the order was rolled back.
That single query, made possible by structured fields and a shared correlation ID, turns a multi hour cross team investigation into a one minute lookup. This is why logging is treated as core infrastructure and not an afterthought.
Where it is used in production
Elastic (ELK Stack)
Elasticsearch, Logstash, and Kibana are the most widely deployed open source log search and visualization stack.
Grafana Loki
Indexes only log labels instead of full text, which makes it cheaper to run at high volume alongside Grafana dashboards.
Splunk
Enterprise platform used heavily for log analytics, security monitoring, and compliance auditing.
Amazon CloudWatch Logs
AWS managed service that collects logs from EC2, Lambda, and containers for search, alarms, and retention.
Frequently asked questions
- What is the difference between logging and monitoring?
- Logging records individual events with full detail so you can reconstruct exactly what happened. Monitoring watches aggregated metrics over time and alerts when something crosses a threshold. You monitor to know that something is wrong, then read logs to find out why.
- Why are structured logs better than plaintext?
- Structured logs write each event as fields, usually JSON, so you can search, filter, count, and group by any field such as user_id or status_code. Plaintext can only be searched with text matching like grep, which makes querying at scale slow and unreliable.
- What log levels should I use?
- Use DEBUG for detailed diagnostic output during development, INFO for normal events worth recording, WARN for unexpected but recoverable situations, and ERROR for failures that need attention. In production, run at INFO by default and enable DEBUG only while investigating an issue.
- Why ship logs to a central system instead of reading them on the server?
- In a distributed system a single request touches many machines, and containers are short lived so their local logs disappear when they restart. Shipping logs to a central store lets you search across all services at once, correlate events with a request ID, and keep history for auditing.
- What should never go into logs?
- Passwords, API keys, session tokens, full credit card numbers, and personal data. Logs are usually readable by far more people than your database, so leaking secrets into them is a common security and privacy breach. Scrub, mask, or hash sensitive fields before writing.
Learn Logging hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Logging as part of a larger topic.
Write-Ahead Logging
Write changes to a sequential log before applying them, the foundation of database crash recovery
advanced · consistency models
Audit Logging
Record who did what, when, and why, the immutable trail that compliance, security, and debugging all depend on
intermediate · data governance compliance
Centralized Logging
One place for all logs, the operational backbone of distributed systems
intermediate · observability monitoring
Structured Logging
Logs as data, not text, making every log line queryable and machine-parseable
intermediate · observability monitoring
Security Audit Logging
Recording security-relevant events with tamper-proof, queryable audit trails for compliance and incident investigation
intermediate · security architecture
See also
Related glossary terms you might want to look up next.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Elasticsearch
A distributed search and analytics engine built on Apache Lucene. Powers full-text search, log analysis, and real-time analytics at scale.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.