We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
In short
Observability is the practice of being able to figure out what a running system is doing internally just by looking at the data it emits, without shipping new code to ask new questions. It rests on three kinds of telemetry: metrics (numbers over time), logs (timestamped event records), and traces (the path of a single request across services).
In short
Observability is the practice of being able to figure out what a running system is doing internally just by looking at the data it emits, without shipping new code to ask new questions. It rests on three kinds of telemetry: metrics (numbers over time), logs (timestamped event records), and traces (the path of a single request across services).
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Observability across a run of lessons, not one page. These 4 alone are about 55 minutes of step-by-step reading, every one with a quiz.
The three pillars of observability, logs, metrics, and traces, and how they work together to make complex systems understandable.
The vendor-neutral observability framework, one standard for traces, metrics, and logs across every language and platform.
The all-in-one observability platform, metrics, logs, traces, and more in a single SaaS product.
The original APM pioneer, now a full observability platform with a generous free tier.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Observability Overviewpay once,
yours for life
The word comes from control theory, where a system is observable if you can infer its internal state from its outputs. Applied to software, it means you can answer questions about why a service is slow, why a request failed, or why memory is climbing, using the telemetry the system already produces. The test of a truly observable system is that you can debug a brand new, unexpected problem without adding new instrumentation first.
This is the difference people draw between monitoring and observability. Monitoring answers questions you knew to ask ahead of time: is CPU above 80 percent, is the error rate over 1 percent. Observability is about the questions you did not predict, like why do checkout requests from one specific mobile app version time out only between 2pm and 3pm. Monitoring tells you that something is wrong. Observability helps you find out why.
In practice teams talk about three pillars: metrics, logs, and traces. They overlap and reinforce each other, and modern tooling increasingly stitches them together so you can jump from a spiking metric to the exact log lines and the exact trace that caused it.
Metrics are numeric measurements aggregated over time, like request count, p99 latency, or queue depth. They are cheap to store because they are just numbers in time buckets, so you can keep months of them and run fast queries and alerts. The cost is that they lose detail: a counter tells you 500 errors happened, not which user or which request caused them.
Logs are discrete, timestamped records of events, ideally structured as key value pairs (JSON) rather than free text so machines can search and group them. They carry rich context per event but are expensive at volume, which is why high traffic systems sample or rate limit them. A single Google or Netflix scale service can produce terabytes of logs per day.
Traces follow one request as it hops across services. Each unit of work is a span, and spans are linked by a shared trace ID and parent span IDs so the full call tree can be reconstructed. A trace shows you that a 900ms checkout spent 600ms waiting on a downstream inventory call, which a metric alone would never reveal. The open standard for emitting all three is OpenTelemetry, which most vendors now ingest.
The glue is context propagation. A trace ID and request attributes are carried through HTTP headers and message metadata so that the metric, the log line, and the span for the same request can be correlated after the fact.
Observability earns its keep once you run distributed systems where a single user action touches many services. In a single monolith, attaching a debugger and reading a stack trace often suffices. In microservices, the failure is rarely in one place, so you need to see the request flow across boundaries. The more services and the higher the request volume, the more you need traces and structured logs, not just dashboards.
The main trade-off is cost and noise. Capturing every log and every trace at full fidelity is expensive in storage and in the vendor bills, which is why teams use sampling. Head based sampling keeps a fixed percentage of traces decided at the start; tail based sampling keeps traces only if they are slow or errored, which is smarter but needs a buffer to hold spans until the request finishes. Either way you accept that you might miss the one trace you wanted.
There is also instrumentation overhead. Adding spans and structured logging costs developer time and a small amount of runtime CPU and memory. Cardinality is the sharpest pitfall: tagging a metric with something unbounded like user ID or request URL can explode the number of time series into the millions and bankrupt your metrics backend. The discipline is to keep high cardinality data in logs and traces, and keep metrics low cardinality.
Imagine a checkout request that normally takes 200ms suddenly shows a p99 of 3 seconds on a dashboard. That spike is a metric. You click into it and the tool surfaces the traces from that window, and you see that 90 percent of the slow traces have one span in common: a call to the payments service that is taking 2.8 seconds.
You open the logs filtered by that trace ID and find repeated lines saying the database connection pool is exhausted. Now you have the chain: a metric told you something was wrong, a trace told you where, and logs told you why. That round trip from symptom to root cause, in minutes rather than hours, is exactly what observability is for.
This is how teams at companies running thousands of services actually operate. Uber, for example, built distributed tracing precisely because no human could reason about a request crossing dozens of microservices from logs alone.
Prometheus
The de facto open source metrics system; scrapes numeric time series and powers alerting, originally built at SoundCloud and now a CNCF project.
Grafana
Visualizes metrics, logs (via Loki), and traces (via Tempo) in one place, the dashboard layer most teams put on top of their telemetry.
Jaeger
Open source distributed tracing system created at Uber to follow requests across hundreds of microservices and find latency bottlenecks.
Datadog
A commercial platform that unifies metrics, logs, traces, and alerting, widely used by teams that prefer a managed service over self hosting.
Monitoring watches for known problems using predefined dashboards and alerts, like CPU over 80 percent. Observability lets you investigate unknown problems after the fact using metrics, logs, and traces, so you can answer questions you did not anticipate without shipping new code. Monitoring tells you something broke; observability helps you find out why.
Metrics (aggregated numbers over time, cheap to store, good for alerts), logs (detailed timestamped event records, rich but expensive at volume), and traces (the path of a single request across services, showing where time was spent). They are most useful when correlated through a shared trace ID.
Usually not. For a single service or monolith, structured logs and a debugger cover most cases. Tracing pays off once a request crosses multiple services, because that is when you cannot tell from logs alone which service caused the slowdown or failure.
OpenTelemetry is a vendor neutral open standard and set of libraries for generating and exporting metrics, logs, and traces. Instrumenting once with OpenTelemetry lets you send the same telemetry to many backends like Prometheus, Jaeger, or Datadog without rewriting your code.
Each unique combination of metric labels creates a separate time series. Tagging a metric with an unbounded value like user ID or full URL can produce millions of series, which slows queries and blows up storage costs. Keep metric labels low cardinality and push detailed per request data into logs and traces instead.
OpenTelemetry
The vendor-neutral observability framework, one standard for traces, metrics, and logs across every language and platform
Datadog
The all-in-one observability platform, metrics, logs, traces, and more in a single SaaS product
New Relic
The original APM pioneer, now a full observability platform with a generous free tier
Dynatrace
AI-driven full-stack observability, automatic discovery, baselining, and root cause analysis
Why Evals Are the Job: Moving From Vibes to Measurement
Why evals, not observability, decide if your LLM feature is good, and why measuring quality is the core skill of AI engineering in 2026.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Logging
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
Log Aggregation
Collecting logs from all services into a central searchable store. The ELK stack (Elasticsearch, Logstash, Kibana) and Grafana Loki are common solutions.
This page explains the idea. The "Observability Overview" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Observability Overview lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Observability in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
