Distributed Tracing

In a monolith, when a request is slow you read one log file and one stack trace. In a system built from 30 microservices, a single click might touch an API gateway, an auth service, a product service, a pricing service, a cache, and three databases. A normal log line tells you that one service was slow, but not which downstream call caused it. Distributed tracing fixes that by stitching all of those steps into one record.

A trace represents the full journey of one request. It is made up of spans. A span is one unit of work, like an HTTP call, a database query, or a function execution. Each span has a start time, a duration, a name, and a set of tags such as the HTTP status code or the SQL statement. Spans link to a parent span, so the whole trace forms a tree that mirrors how services called each other.

The piece that makes it work across machines is context propagation. The first service generates a trace ID and a span ID, then passes them in the request headers to the next service. Every service reuses the same trace ID and creates a new span underneath. When all the spans are sent to a central backend and joined on the trace ID, you get one connected picture instead of dozens of disconnected logs.

Tracing libraries instrument your code so spans are created automatically around common operations: incoming HTTP requests, outgoing HTTP calls, database queries, and message queue publishes. Most modern setups use OpenTelemetry, an open standard that defines the data format and the propagation headers so a Java service and a Go service can share one trace.

Context travels over the wire in a header. The W3C Trace Context standard uses a header called traceparent that carries the trace ID, the parent span ID, and a sampled flag. An older but still common format is Zipkin B3, which splits the same data across X-B3-TraceId and related headers. As long as both sides agree on the format, the trace stays connected across service and language boundaries.

Each service exports its finished spans to a collector, often an OpenTelemetry Collector, which forwards them to a backend like Jaeger, Tempo, or a vendor such as Datadog. The backend indexes spans by trace ID and reassembles the tree. Because tracing every single request at high traffic is expensive, systems apply sampling. Head sampling decides at the start to keep, say, 1 percent of traces. Tail sampling waits until the trace is complete and keeps it only if it was slow or contained an error, which captures the interesting traces without storing everything.

Reach for distributed tracing when a request crosses service boundaries and you need to answer latency and failure questions. It is the fastest way to find which of ten downstream calls added 400 milliseconds, or which service in a chain returned the 500 that the user saw. Logs tell you what happened inside one service and metrics tell you aggregate health, but only traces show the causal path of one request end to end.

The main cost is instrumentation and data volume. You need every service to propagate context, otherwise the trace breaks at the first service that drops the headers. Auto-instrumentation handles most of this, but custom spans for important business logic still take effort. Stored span data is also large, which is why sampling matters; keeping 100 percent of traces at millions of requests per second is rarely worth the storage bill.

Tracing complements logs and metrics rather than replacing them. The common pattern is to attach the trace ID to your log lines so you can jump from a slow trace straight to the detailed logs for that exact request. Treat tracing as the map that tells you where to look, then use logs for the fine detail.

Say a user reports that checkout takes four seconds. You open the trace for that request and see the root span: a POST to the checkout endpoint lasting 4.1 seconds. Underneath it are child spans for the auth check at 20 milliseconds, the cart service at 30 milliseconds, and the payment service at 3.9 seconds.

Expanding the payment span, you see it made three sequential calls to a fraud-scoring service, each taking about 1.3 seconds. The bottleneck is clear: the calls run one after another instead of in parallel, and the fraud service itself is slow. Without tracing you would have guessed at logs across four services; with it, the answer is visible in one screen in under a minute.

This is the everyday payoff. The trace turns a vague complaint into a specific, fixable line of code, and it points at the exact service and call that owns the latency.