Observability

The word comes from control theory, where a system is observable if you can infer its internal state from its outputs. Applied to software, it means you can answer questions about why a service is slow, why a request failed, or why memory is climbing, using the telemetry the system already produces. The test of a truly observable system is that you can debug a brand new, unexpected problem without adding new instrumentation first.

This is the difference people draw between monitoring and observability. Monitoring answers questions you knew to ask ahead of time: is CPU above 80 percent, is the error rate over 1 percent. Observability is about the questions you did not predict, like why do checkout requests from one specific mobile app version time out only between 2pm and 3pm. Monitoring tells you that something is wrong. Observability helps you find out why.

In practice teams talk about three pillars: metrics, logs, and traces. They overlap and reinforce each other, and modern tooling increasingly stitches them together so you can jump from a spiking metric to the exact log lines and the exact trace that caused it.

Metrics are numeric measurements aggregated over time, like request count, p99 latency, or queue depth. They are cheap to store because they are just numbers in time buckets, so you can keep months of them and run fast queries and alerts. The cost is that they lose detail: a counter tells you 500 errors happened, not which user or which request caused them.

Logs are discrete, timestamped records of events, ideally structured as key value pairs (JSON) rather than free text so machines can search and group them. They carry rich context per event but are expensive at volume, which is why high traffic systems sample or rate limit them. A single Google or Netflix scale service can produce terabytes of logs per day.

Traces follow one request as it hops across services. Each unit of work is a span, and spans are linked by a shared trace ID and parent span IDs so the full call tree can be reconstructed. A trace shows you that a 900ms checkout spent 600ms waiting on a downstream inventory call, which a metric alone would never reveal. The open standard for emitting all three is OpenTelemetry, which most vendors now ingest.

The glue is context propagation. A trace ID and request attributes are carried through HTTP headers and message metadata so that the metric, the log line, and the span for the same request can be correlated after the fact.

Observability earns its keep once you run distributed systems where a single user action touches many services. In a single monolith, attaching a debugger and reading a stack trace often suffices. In microservices, the failure is rarely in one place, so you need to see the request flow across boundaries. The more services and the higher the request volume, the more you need traces and structured logs, not just dashboards.

The main trade-off is cost and noise. Capturing every log and every trace at full fidelity is expensive in storage and in the vendor bills, which is why teams use sampling. Head based sampling keeps a fixed percentage of traces decided at the start; tail based sampling keeps traces only if they are slow or errored, which is smarter but needs a buffer to hold spans until the request finishes. Either way you accept that you might miss the one trace you wanted.

There is also instrumentation overhead. Adding spans and structured logging costs developer time and a small amount of runtime CPU and memory. Cardinality is the sharpest pitfall: tagging a metric with something unbounded like user ID or request URL can explode the number of time series into the millions and bankrupt your metrics backend. The discipline is to keep high cardinality data in logs and traces, and keep metrics low cardinality.

Imagine a checkout request that normally takes 200ms suddenly shows a p99 of 3 seconds on a dashboard. That spike is a metric. You click into it and the tool surfaces the traces from that window, and you see that 90 percent of the slow traces have one span in common: a call to the payments service that is taking 2.8 seconds.

You open the logs filtered by that trace ID and find repeated lines saying the database connection pool is exhausted. Now you have the chain: a metric told you something was wrong, a trace told you where, and logs told you why. That round trip from symptom to root cause, in minutes rather than hours, is exactly what observability is for.

This is how teams at companies running thousands of services actually operate. Uber, for example, built distributed tracing precisely because no human could reason about a request crossing dozens of microservices from logs alone.