We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
In short
Distributed tracing is a way to follow a single request as it travels through every service in a distributed system, recording the timing and outcome of each step. Each service attaches a shared trace ID and its own span, so you end up with one connected timeline that shows exactly where the request went and where it slowed down or failed.
In short
Distributed tracing is a way to follow a single request as it travels through every service in a distributed system, recording the timing and outcome of each step. Each service attaches a shared trace ID and its own span, so you end up with one connected timeline that shows exactly where the request went and where it slowed down or failed.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Distributed Tracing across a run of lessons, not one page. These 4 alone are about 70 minutes of step-by-step reading, every one with a quiz.
End-to-end request visibility across microservices, seeing the full journey from entry to response.
Passing trace context, baggage, and metadata across service boundaries, the glue of distributed tracing.
The unique identifier for a distributed trace, one ID that ties together every service a request touches.
Individual operations within a trace, each span is one unit of work with a start time, duration, and parent.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Distributed Tracingpay once,
yours for life
In a monolith, when a request is slow you read one log file and one stack trace. In a system built from 30 microservices, a single click might touch an API gateway, an auth service, a product service, a pricing service, a cache, and three databases. A normal log line tells you that one service was slow, but not which downstream call caused it. Distributed tracing fixes that by stitching all of those steps into one record.
A trace represents the full journey of one request. It is made up of spans. A span is one unit of work, like an HTTP call, a database query, or a function execution. Each span has a start time, a duration, a name, and a set of tags such as the HTTP status code or the SQL statement. Spans link to a parent span, so the whole trace forms a tree that mirrors how services called each other.
The piece that makes it work across machines is context propagation. The first service generates a trace ID and a span ID, then passes them in the request headers to the next service. Every service reuses the same trace ID and creates a new span underneath. When all the spans are sent to a central backend and joined on the trace ID, you get one connected picture instead of dozens of disconnected logs.
Tracing libraries instrument your code so spans are created automatically around common operations: incoming HTTP requests, outgoing HTTP calls, database queries, and message queue publishes. Most modern setups use OpenTelemetry, an open standard that defines the data format and the propagation headers so a Java service and a Go service can share one trace.
Context travels over the wire in a header. The W3C Trace Context standard uses a header called traceparent that carries the trace ID, the parent span ID, and a sampled flag. An older but still common format is Zipkin B3, which splits the same data across X-B3-TraceId and related headers. As long as both sides agree on the format, the trace stays connected across service and language boundaries.
Each service exports its finished spans to a collector, often an OpenTelemetry Collector, which forwards them to a backend like Jaeger, Tempo, or a vendor such as Datadog. The backend indexes spans by trace ID and reassembles the tree. Because tracing every single request at high traffic is expensive, systems apply sampling. Head sampling decides at the start to keep, say, 1 percent of traces. Tail sampling waits until the trace is complete and keeps it only if it was slow or contained an error, which captures the interesting traces without storing everything.
Reach for distributed tracing when a request crosses service boundaries and you need to answer latency and failure questions. It is the fastest way to find which of ten downstream calls added 400 milliseconds, or which service in a chain returned the 500 that the user saw. Logs tell you what happened inside one service and metrics tell you aggregate health, but only traces show the causal path of one request end to end.
The main cost is instrumentation and data volume. You need every service to propagate context, otherwise the trace breaks at the first service that drops the headers. Auto-instrumentation handles most of this, but custom spans for important business logic still take effort. Stored span data is also large, which is why sampling matters; keeping 100 percent of traces at millions of requests per second is rarely worth the storage bill.
Tracing complements logs and metrics rather than replacing them. The common pattern is to attach the trace ID to your log lines so you can jump from a slow trace straight to the detailed logs for that exact request. Treat tracing as the map that tells you where to look, then use logs for the fine detail.
Say a user reports that checkout takes four seconds. You open the trace for that request and see the root span: a POST to the checkout endpoint lasting 4.1 seconds. Underneath it are child spans for the auth check at 20 milliseconds, the cart service at 30 milliseconds, and the payment service at 3.9 seconds.
Expanding the payment span, you see it made three sequential calls to a fraud-scoring service, each taking about 1.3 seconds. The bottleneck is clear: the calls run one after another instead of in parallel, and the fraud service itself is slow. Without tracing you would have guessed at logs across four services; with it, the answer is visible in one screen in under a minute.
This is the everyday payoff. The trace turns a vague complaint into a specific, fixable line of code, and it points at the exact service and call that owns the latency.
Google Dapper
The 2010 paper that started modern distributed tracing; it traced requests across Google's internal services and inspired Zipkin and Jaeger.
Uber Jaeger
Uber built Jaeger to trace its thousands of microservices, then open sourced it; it is now a CNCF project and a default tracing backend.
Netflix
Traces requests across its microservice fleet to find latency hotspots and pinpoint which downstream service degraded a stream start.
Datadog
Offers a hosted APM product that ingests OpenTelemetry spans and links traces to logs and metrics for end-to-end request views.
A span is one unit of work, like a single HTTP call or database query, with a start time, duration, and tags. A trace is the full collection of spans for one request, joined by a shared trace ID and arranged into a parent-child tree.
Logs are independent records from inside one service with no built-in link between services. Tracing connects every step of a single request across all services into one timeline, showing the causal path and the latency of each hop. They work best together: put the trace ID in your logs so you can jump from a trace to the detailed logs.
OpenTelemetry is an open standard and set of SDKs for generating traces, metrics, and logs. It defines the span data format and the context propagation headers, so services written in different languages can produce one connected trace and send it to any compatible backend like Jaeger, Tempo, or Datadog.
Recording and storing a trace for every request at high traffic is expensive in compute and storage. Sampling keeps a subset. Head sampling decides up front to keep a fixed percentage, while tail sampling waits until the trace finishes and keeps it only if it was slow or errored, capturing the useful traces without storing everything.
Through context propagation. The first service creates a trace ID and passes it to the next service in a request header, usually the W3C traceparent header or the older B3 headers. Each service reuses that trace ID and adds its own span, so the backend can join all spans on the trace ID.
Distributed Context Propagation
Passing trace context, baggage, and metadata across service boundaries, the glue of distributed tracing
Trace IDs
The unique identifier for a distributed trace, one ID that ties together every service a request touches
Span IDs
Individual operations within a trace, each span is one unit of work with a start time, duration, and parent
Jaeger
Uber's open-source distributed tracing system, built for microservices at massive scale
Zipkin
Twitter's original distributed tracing system, the pioneer that inspired modern tracing
Microservices
An architecture where an application is split into small, independent services that communicate over the network. Each service owns its own data and can be deployed separately.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Logging
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
This page explains the idea. The "Distributed Tracing" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Distributed Tracing lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Distributed Tracing in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
