Distributed Tracing

A trading platform experienced intermittent order failures, about 2% of orders silently failed with no error in any individual service's logs. Each service reported "I did my job correctly." The order service processed the order. The inventory service reserved the stock. The payment service charged the card. The notification service sent the confirmation. But 2% of customers never received their products.

Six engineers spent three days grepping logs, correlating timestamps, and building spreadsheets mapping request flows. On day three, they found it: the fulfillment service was occasionally receiving a stale inventory snapshot due to replication lag. It shipped the wrong items, triggering a return, which the returns service processed without flagging the original order as problematic.

With distributed tracing, this would have taken 15 minutes. Pull up a failed order's trace, see every service call in sequence, spot the stale data in the fulfillment service's span attributes, correlate with the replication lag visible in the inventory service's database span.

Distributed tracing is the single most impactful observability investment for microservices architectures. It provides a unified view of a request's journey through all services, something logs and metrics alone cannot.

A distributed trace is a tree of spans representing every operation in a request's lifecycle:

1. Trace starts, the first instrumented service (API gateway) creates a root span and generates a trace ID
2. Context propagates, each outbound call carries the trace ID and parent span ID in headers
3. Spans are created, each service creates spans for its operations (HTTP handlers, DB queries, external calls)
4. Spans are reported, each service sends its spans to a trace collector (asynchronously, not blocking the request)
5. Trace is assembled, the collector receives spans from all services and reconstructs the full trace tree using trace ID and parent-child relationships
6. Trace is queryable, engineers search by trace ID, service, duration, error status, or any span attribute