Name: System Design Masterclass
Availability: InStock

A user on Twitter changes their display name from "John" to "Johnny." They refresh their profile page. It still says "John." They refresh again. Still "John." Three minutes later, it finally shows "Johnny."

What happened? The write went to Twitter's primary database. The profile page was served by a read replica. And that replica was three minutes behind the primary. The user's update existed in the system. It just hadn't propagated to the replica that served the read request yet.

This is lag, and it's one of the most common sources of user-visible bugs in distributed systems. Every company that uses read replicas (which is basically every company at scale) has fought this battle. GitHub, Slack, Shopify, LinkedIn, they've all had outages or bug reports traced back to replication lag.

The frustrating part? The system is working exactly as designed. Replication lag isn't a bug. It's a fundamental trade-off of distributed databases.

A decay curve showing how often a read misses its own write, plotted against the gap between writing and reading at sampled gaps of 0, 10, 25, 50, 100, 200 and 400 milliseconds. Reading any follower at random misses the write 100 percent of the time at a zero gap, 98.8 percent at 10 ms, 89.5 at 25, 62.4 at 50, 39.9 at 100, 16.0 at 200 and 3.5 at 400, for 1,407 stale reads out of 2,400. Sticking to the primary for 300 milliseconds after a write gives zero stale reads until the 400 ms gap, where it falls back to a follower and reaches 2.9 percent, for 10 stale of 2,400. Reading the primary only gives zero stale reads throughout. Dashed lines mark the measured median replication lag of 67.6 ms and the p95 of 394.4 ms. — Chart2,400 writes, each read back, on real threads with real queues
The Twitter story above is the extreme version. Here is the ordinary one, measured.
Look at the leftmost point on the red line. A user writes, then immediately reads, and lands on a follower. They miss their own write 100 percent of the time. Not occasionally. Every single time, because no replica has had a chance to hear about it yet.
The curve only starts to fall once the wait passes the replication lag, which is why the median lag line sits right where the steep part is. Before that line you are asking a replica about something it has not been told. After it, you have better than even odds.
That is what makes this so hard to catch in testing. Your test writes and reads back in the same millisecond, so it fails immediately and you notice. A real user clicks Save, the page navigates, and 200 milliseconds pass on their behalf, so they only see it sometimes. The bug you cannot reproduce is the bug that reaches production.
The blue line is the fix that costs you least. Pin a client to the primary for 300 milliseconds after it writes and let it go back to followers afterwards. Ten stale reads out of 2,400 instead of 1,407, and your read fleet still carries almost all the traffic. The green line, reading the primary for everything, has no stale reads at all and no read scaling either, which is the thing you bought replicas for.
One number here deserves suspicion in your own system. The 67.6 ms median came from a lab where one follower was made four times slower than the others. Yours will be different, it will vary by hour, and it will get much worse during exactly the incidents when you need it to behave. Measure it before choosing a sticky window, and alert on it.

Replication Lag

The rest of this lesson

Continue this lesson

The Update That Disappeared