Clock Skew
The difference in time readings between clocks on different machines. Physical clocks drift; NTP helps but can't guarantee perfect sync. Why distributed systems use logical clocks.
What is Clock Skew?
In short
Clock skew is the difference in time that two machines report at the same instant, caused by their hardware clocks ticking at slightly different rates. Even with NTP keeping them roughly aligned, server clocks in a real data center commonly disagree by a few milliseconds to tens of milliseconds, which is why distributed systems cannot safely order events by comparing wall-clock timestamps alone.
What clock skew actually is
Every computer keeps time with a quartz crystal oscillator. The crystal vibrates at a nominal frequency, and the OS counts those vibrations to advance the clock. The problem is that no two crystals are identical. Temperature, voltage, and manufacturing tolerances make each one run a little fast or a little slow. That rate error is called clock drift, and it is usually quoted as parts per million.
A typical server crystal drifts by 10 to 50 ppm. At 50 ppm a clock gains or loses about 50 microseconds every second, which works out to roughly 4 seconds per day if nothing corrects it. Two machines drifting in opposite directions diverge twice as fast. The gap between what they read at the same moment is the clock skew.
Skew is the standing difference between clocks; drift is the rate at which that difference grows. You correct drift by re-syncing, but the act of measuring time over a network introduces its own uncertainty, so skew never truly hits zero.
How systems try to control it, and why they cannot eliminate it
NTP, the Network Time Protocol, is the standard fix. A machine asks a time server for the current time, measures the round trip, and assumes the true time is the server reading plus half the round trip. It then steps or slews its local clock toward that value. Public NTP keeps machines within about 1 to 50 milliseconds of each other; well tuned hardware on a local network can reach sub-millisecond.
The flaw is in the half-round-trip assumption. Network paths are not symmetric, packets queue, and the request leg may take longer than the reply leg. That asymmetry shows up directly as residual skew. PTP, the Precision Time Protocol, plus hardware timestamping on the network card pushes accuracy into the microsecond range, but it needs switch support and still leaves a nonzero bound.
A subtle hazard is that NTP can step a clock backward when it corrects a fast machine. Code that assumes time only moves forward can see a timestamp from the future, then a smaller one, and break. This is why you read monotonic clocks for measuring durations and the wall clock only for displaying or stamping calendar time.
When it bites, and what to use instead
Skew breaks any design that orders events across machines by comparing wall-clock timestamps. Last-write-wins conflict resolution is the classic trap: if node A and B both write a key and B's clock is 30 ms behind, A's later write can be discarded because its timestamp looks older. Distributed locks with timeouts, certificate expiry checks, and TTL based caches all misbehave when the clocks involved disagree.
The standard answer is logical clocks, which order events by causality instead of physical time. A Lamport clock is a counter that increments on every event and travels with each message, giving a consistent total order. Vector clocks go further and detect whether two events are concurrent, which is what lets systems like DynamoDB flag write conflicts instead of silently losing data.
When you genuinely need physical time for ordering, you bound the uncertainty and design around it. The rule of thumb is simple: if the difference between two timestamps is smaller than your worst-case skew, you cannot trust which one came first, so either fall back to a logical tiebreaker or wait out the uncertainty window.
A concrete example: Google Spanner and TrueTime
Google Spanner needs globally consistent transaction ordering across data centers on different continents, which is exactly the case where skew is worst. Instead of pretending clocks are perfect, Spanner exposes the uncertainty directly through an API called TrueTime.
TrueTime does not return a single timestamp. It returns an interval, earliest and latest, that is guaranteed to contain the true time. Google backs this with GPS receivers and atomic clocks in every data center, which keeps the interval width down to roughly 1 to 7 milliseconds.
To commit a transaction, Spanner picks a timestamp and then deliberately waits until TrueTime guarantees that instant has passed everywhere, a step called commit wait. By paying that few milliseconds of latency it makes skew irrelevant to correctness: any transaction that starts after another commits is guaranteed a later timestamp. It is a clean demonstration of the core lesson, which is that you handle clock skew by measuring and bounding it, not by assuming it away.
Where it is used in production
Google Spanner
TrueTime returns a time interval with a bounded error and uses commit wait so global transactions get correct timestamp ordering despite cross-continent skew.
Amazon DynamoDB
Uses vector clocks to detect concurrent writes rather than trusting wall-clock timestamps, then surfaces conflicting versions for resolution.
Apache Cassandra
Last-write-wins by timestamp means a skewed node clock can silently drop newer data, so operators run tight NTP and avoid time-sensitive conflict patterns.
Apache Kafka
Log retention and time-based offset lookups depend on broker and producer clocks, so message timestamp semantics and NTP discipline matter for correct expiry.
Frequently asked questions
- What is the difference between clock skew and clock drift?
- Skew is the current difference between two clocks at the same instant. Drift is the rate at which that difference grows because the clocks tick at slightly different speeds. Drift causes skew to accumulate; re-syncing with NTP resets the skew but does nothing about the underlying drift.
- Does NTP eliminate clock skew?
- No. NTP reduces skew to roughly 1 to 50 milliseconds on the public internet and can reach sub-millisecond on a local network, but network path asymmetry and queuing leave a nonzero residual error. PTP with hardware timestamping gets to microseconds but still cannot reach exactly zero.
- Why do distributed systems use logical clocks instead of wall-clock time?
- Because wall clocks on different machines disagree, ordering events by comparing timestamps is unreliable. Logical clocks like Lamport and vector clocks order events by causality using counters that travel with messages, so they give a consistent order without depending on any physical clock being accurate.
- How does clock skew cause data loss in last-write-wins systems?
- In last-write-wins, the write with the higher timestamp wins. If one node's clock runs behind, a genuinely newer write from that node gets a smaller timestamp than an older write elsewhere, so the system keeps the stale value and discards the fresh one. The data loss is silent.
- Should I use the wall clock or a monotonic clock to measure elapsed time?
- Use a monotonic clock for durations. The wall clock can jump backward when NTP corrects it, which produces negative or wildly wrong elapsed times. Reserve the wall clock for calendar timestamps you display or store, and use the monotonic clock for any timing or timeout logic.
Learn Clock Skew hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
See also
Related glossary terms you might want to look up next.
Lamport Timestamp
A simple logical clock where each event increments a counter. If event A causes event B, A's timestamp is always less than B's. The foundation of logical time in distributed systems.
Vector Clock
A logical clock that tracks causality across distributed nodes using a vector of counters. Each node increments its own counter and merges vectors on message receipt.
Distributed Lock
A lock that coordinates access to a shared resource across multiple machines. Implemented via Redis (Redlock), ZooKeeper, or etcd. Much harder than local locks.
CAP Theorem
In a distributed system, you can only guarantee two of three: Consistency, Availability, and Partition tolerance. You must choose your trade-off.
Consensus
The process of getting multiple nodes in a distributed system to agree on a single value. The foundation of distributed databases and coordination services.
Paxos
A family of protocols for solving consensus in unreliable networks. Famously difficult to understand but mathematically proven correct.