Trace
The full end-to-end path of a request through a distributed system, composed of multiple spans. A trace shows which services were called, in what order, and how long each took.
What is Trace?
In short
A trace is the full end-to-end record of one request as it moves through a distributed system, stitched together from many smaller timed units called spans. It shows every service the request touched, the order they were called in, and how long each step took, so you can see exactly where time went and where things failed.
What a trace actually is
When a user clicks a button, that single action often fans out into dozens of internal calls: an API gateway, an auth service, a product service, a database, a cache, a payment provider. A trace is the complete story of that one request across all of those hops, assembled into a single timeline.
A trace is made of spans. Each span represents one unit of work, for example the database query, the call to the auth service, or the time spent in the gateway. Every span records a start time, a duration, the service that produced it, and key/value attributes like the HTTP status code or the SQL statement. Spans nest: a parent span (the gateway handling the request) contains child spans (the downstream calls it made).
What ties the spans together is a shared trace ID. The first service that receives the request generates the trace ID and a root span ID, then passes both forward in request headers. Every downstream service reuses the same trace ID and adds its own spans pointing back to the parent. When you collect all spans that share a trace ID, you get the full tree.
How context propagation works under the hood
The mechanism that makes distributed tracing possible is context propagation. The trace ID and parent span ID travel with the request, usually in an HTTP header. The current open standard is W3C Trace Context, which defines a header called traceparent that looks like 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The four fields are version, the 16-byte trace ID, the 8-byte parent span ID, and trace flags such as the sampled bit.
Each service instruments its inbound and outbound calls. On the way in, it reads traceparent and starts a child span. On the way out to the next service, it writes a fresh traceparent with its own span as the new parent. Libraries like OpenTelemetry do this automatically for common frameworks, so you rarely write the propagation code by hand.
Spans are not sent home one at a time. Each service buffers spans and ships them in batches to a collector, which forwards them to a backend like Jaeger, Tempo, or a commercial tool. The backend groups spans by trace ID and reconstructs the waterfall view you look at in the UI. Because spans arrive independently and out of order, the backend tolerates partial traces and assembles whatever it receives.
When to use tracing and the trade-offs
Reach for tracing when a request crosses service boundaries and you need to answer where is the latency or which service threw the error. Logs tell you what happened inside one process, and metrics tell you aggregate rates and percentiles, but only a trace shows the causal path of a single request through many processes. It is the fastest way to find the one slow database call buried inside a 2 second checkout.
The main cost is volume. A busy system can produce billions of spans a day, and storing all of them is expensive. The standard answer is sampling. Head-based sampling decides at the start of the request whether to keep it, for example keep 1 percent. Tail-based sampling waits until the trace is complete and then keeps it only if it was slow or errored, which is more useful but needs more memory in the collector.
The other cost is instrumentation effort and overhead. Auto-instrumentation covers popular libraries, but custom business logic still needs manual spans to be visible. Done carelessly, tracing adds CPU and a little latency per span. In practice the per-span overhead is small, on the order of microseconds, and sampling keeps the storage bill in check.
A concrete example
Picture an e-commerce checkout that suddenly feels slow. You open the trace for one slow request and see a waterfall: the gateway span is 1,800 ms, inside it the cart service is 120 ms, the inventory service is 90 ms, and the payment service span is 1,500 ms. The payment span has a child span for an external HTTP call to the payment provider that took 1,450 ms.
Without tracing you would only know the request was slow. With the trace you can see instantly that the bottleneck is an outbound call to the payment gateway, not your own code or your database. You stop guessing and go straight to adding a timeout, a retry budget, or a circuit breaker around that one call.
This is why teams running microservices treat tracing as a core debugging tool. The trace ID also acts as a join key. You can copy it from an error log line, paste it into the tracing UI, and immediately see the whole request that produced that log, which collapses an investigation that used to take an hour into a couple of minutes.
Where it is used in production
Google Dapper
The original large-scale distributed tracing system, described in Google's 2010 Dapper paper, which inspired nearly every tracing tool that followed.
Jaeger
Open-source tracing backend started at Uber, now a CNCF project, used to store and visualize trace waterfalls from OpenTelemetry data.
Grafana Tempo
A high-volume trace store designed to keep tracing cheap by indexing only the trace ID and relying on object storage like S3.
Datadog APM
Commercial platform that auto-instruments services and ties traces to logs and metrics so one trace ID links all three signals.
Frequently asked questions
- What is the difference between a trace and a span?
- A span is one timed unit of work inside a single service, like one database query. A trace is the whole collection of spans that share the same trace ID, representing one full request across all services. One trace usually contains many spans.
- What is the difference between tracing, logging, and metrics?
- These are the three pillars of observability. Metrics are aggregated numbers like requests per second and p99 latency. Logs are discrete text records of events inside one process. Traces follow a single request across many processes and show the causal path and timing. You usually need all three.
- How does a trace ID get passed between services?
- Through context propagation. The trace ID and parent span ID are carried in request headers, most commonly the W3C traceparent header for HTTP. Each service reads the incoming header, adds its own spans, and writes a new header pointing to its span before calling the next service.
- Won't tracing every request slow my system down or cost too much?
- Per-span overhead is tiny, usually microseconds, so latency impact is minimal. The real cost is storage at high volume. Teams control this with sampling, keeping only a fraction of traces (head-based) or keeping only the slow and errored ones (tail-based).
- What is OpenTelemetry and how does it relate to tracing?
- OpenTelemetry is a vendor-neutral standard and set of libraries for generating traces, metrics, and logs. It auto-instruments common frameworks to create spans and handles context propagation for you, then exports the data to any compatible backend like Jaeger, Tempo, or Datadog.
Learn Trace hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Trace as part of a larger topic.
Trace IDs
The unique identifier for a distributed trace, one ID that ties together every service a request touches
intermediate · observability monitoring
Dynatrace
AI-driven full-stack observability, automatic discovery, baselining, and root cause analysis
intermediate · observability monitoring
Observability Overview
The three pillars of observability, logs, metrics, and traces, and how they work together to make complex systems understandable
intermediate · observability monitoring
Span IDs
Individual operations within a trace, each span is one unit of work with a start time, duration, and parent
intermediate · observability monitoring
Distributed Context Propagation
Passing trace context, baggage, and metadata across service boundaries, the glue of distributed tracing
intermediate · observability monitoring
See also
Related glossary terms you might want to look up next.
Span
A single unit of work in a distributed trace, representing one operation (e.g., an HTTP call or database query). Spans are nested to form a trace tree.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Logging
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.