We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
The full end-to-end path of a request through a distributed system, composed of multiple spans. A trace shows which services were called, in what order, and how long each took.
In short
A trace is the full end-to-end record of one request as it moves through a distributed system, stitched together from many smaller timed units called spans. It shows every service the request touched, the order they were called in, and how long each step took, so you can see exactly where time went and where things failed.
In short
A trace is the full end-to-end record of one request as it moves through a distributed system, stitched together from many smaller timed units called spans. It shows every service the request touched, the order they were called in, and how long each step took, so you can see exactly where time went and where things failed.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Trace across a run of lessons, not one page. These 4 alone are about 53 minutes of step-by-step reading, every one with a quiz.
The unique identifier for a distributed trace, one ID that ties together every service a request touches.
AI-driven full-stack observability, automatic discovery, baselining, and root cause analysis.
The three pillars of observability, logs, metrics, and traces, and how they work together to make complex systems understandable.
Individual operations within a trace, each span is one unit of work with a start time, duration, and parent.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Trace IDspay once,
yours for life
When a user clicks a button, that single action often fans out into dozens of internal calls: an API gateway, an auth service, a product service, a database, a cache, a payment provider. A trace is the complete story of that one request across all of those hops, assembled into a single timeline.
A trace is made of spans. Each span represents one unit of work, for example the database query, the call to the auth service, or the time spent in the gateway. Every span records a start time, a duration, the service that produced it, and key/value attributes like the HTTP status code or the SQL statement. Spans nest: a parent span (the gateway handling the request) contains child spans (the downstream calls it made).
What ties the spans together is a shared trace ID. The first service that receives the request generates the trace ID and a root span ID, then passes both forward in request headers. Every downstream service reuses the same trace ID and adds its own spans pointing back to the parent. When you collect all spans that share a trace ID, you get the full tree.
The mechanism that makes distributed tracing possible is context propagation. The trace ID and parent span ID travel with the request, usually in an HTTP header. The current open standard is W3C Trace Context, which defines a header called traceparent that looks like 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The four fields are version, the 16-byte trace ID, the 8-byte parent span ID, and trace flags such as the sampled bit.
Each service instruments its inbound and outbound calls. On the way in, it reads traceparent and starts a child span. On the way out to the next service, it writes a fresh traceparent with its own span as the new parent. Libraries like OpenTelemetry do this automatically for common frameworks, so you rarely write the propagation code by hand.
Spans are not sent home one at a time. Each service buffers spans and ships them in batches to a collector, which forwards them to a backend like Jaeger, Tempo, or a commercial tool. The backend groups spans by trace ID and reconstructs the waterfall view you look at in the UI. Because spans arrive independently and out of order, the backend tolerates partial traces and assembles whatever it receives.
Reach for tracing when a request crosses service boundaries and you need to answer where is the latency or which service threw the error. Logs tell you what happened inside one process, and metrics tell you aggregate rates and percentiles, but only a trace shows the causal path of a single request through many processes. It is the fastest way to find the one slow database call buried inside a 2 second checkout.
The main cost is volume. A busy system can produce billions of spans a day, and storing all of them is expensive. The standard answer is sampling. Head-based sampling decides at the start of the request whether to keep it, for example keep 1 percent. Tail-based sampling waits until the trace is complete and then keeps it only if it was slow or errored, which is more useful but needs more memory in the collector.
The other cost is instrumentation effort and overhead. Auto-instrumentation covers popular libraries, but custom business logic still needs manual spans to be visible. Done carelessly, tracing adds CPU and a little latency per span. In practice the per-span overhead is small, on the order of microseconds, and sampling keeps the storage bill in check.
Picture an e-commerce checkout that suddenly feels slow. You open the trace for one slow request and see a waterfall: the gateway span is 1,800 ms, inside it the cart service is 120 ms, the inventory service is 90 ms, and the payment service span is 1,500 ms. The payment span has a child span for an external HTTP call to the payment provider that took 1,450 ms.
Without tracing you would only know the request was slow. With the trace you can see instantly that the bottleneck is an outbound call to the payment gateway, not your own code or your database. You stop guessing and go straight to adding a timeout, a retry budget, or a circuit breaker around that one call.
This is why teams running microservices treat tracing as a core debugging tool. The trace ID also acts as a join key. You can copy it from an error log line, paste it into the tracing UI, and immediately see the whole request that produced that log, which collapses an investigation that used to take an hour into a couple of minutes.
Google Dapper
The original large-scale distributed tracing system, described in Google's 2010 Dapper paper, which inspired nearly every tracing tool that followed.
Jaeger
Open-source tracing backend started at Uber, now a CNCF project, used to store and visualize trace waterfalls from OpenTelemetry data.
Grafana Tempo
A high-volume trace store designed to keep tracing cheap by indexing only the trace ID and relying on object storage like S3.
Datadog APM
Commercial platform that auto-instruments services and ties traces to logs and metrics so one trace ID links all three signals.
A span is one timed unit of work inside a single service, like one database query. A trace is the whole collection of spans that share the same trace ID, representing one full request across all services. One trace usually contains many spans.
These are the three pillars of observability. Metrics are aggregated numbers like requests per second and p99 latency. Logs are discrete text records of events inside one process. Traces follow a single request across many processes and show the causal path and timing. You usually need all three.
Through context propagation. The trace ID and parent span ID are carried in request headers, most commonly the W3C traceparent header for HTTP. Each service reads the incoming header, adds its own spans, and writes a new header pointing to its span before calling the next service.
Per-span overhead is tiny, usually microseconds, so latency impact is minimal. The real cost is storage at high volume. Teams control this with sampling, keeping only a fraction of traces (head-based) or keeping only the slow and errored ones (tail-based).
OpenTelemetry is a vendor-neutral standard and set of libraries for generating traces, metrics, and logs. It auto-instruments common frameworks to create spans and handles context propagation for you, then exports the data to any compatible backend like Jaeger, Tempo, or Datadog.
Dynatrace
AI-driven full-stack observability, automatic discovery, baselining, and root cause analysis
Which Traces to Read: Sampling Decides What You Can Discover
You will read a few dozen traces out of a hundred thousand. Measured on the lab: the same 40 traces report a failure rate of 25 percent, 100 percent or 86 percent, and the only thing that changed was the rule that picked them.
Observability Overview
The three pillars of observability, logs, metrics, and traces, and how they work together to make complex systems understandable
Theoretical Saturation: How Many Traces Is Enough?
The answer is not a number, it is a shape. The last new failure code arrived at trace 31 of 40, leaving a tail of nine, which just clears the rule. Reshuffle the reading order twenty thousand times and only 22 percent of orders agree.
Span IDs
Individual operations within a trace, each span is one unit of work with a start time, duration, and parent
Span
A single unit of work in a distributed trace, representing one operation (e.g., an HTTP call or database query). Spans are nested to form a trace tree.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Logging
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.
This page explains the idea. The "Trace IDs" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Trace IDs lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Trace in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
