We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Distributed Systems Core
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Distributed Systems Core
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Imagine you're mailing a package to your friend, but they're on vacation. The postal carrier doesn't throw the package away. They don't return it to you. They hold it at the local post office with a note. "deliver to 42 Elm Street when occupant returns." When your friend gets home, the package is waiting.
Hinted handoff is the same idea applied to distributed databases. When a write needs to go to node B but node B is temporarily down, the write goes to node C instead, with a "hint" that says "this belongs to node B, forward it when B comes back." Node C stores the data temporarily, monitors B's status, and delivers the data once B recovers.
This mechanism was introduced in Amazon's Dynamo paper (2007) and is now used by Cassandra, Riak, and Voldemort. It's the reason these databases can maintain write availability even when individual nodes fail, writes don't get rejected just because one replica is temporarily unreachable.
The key word is "temporarily." Hinted handoff is designed for transient failures, a node rebooting, a brief network issue, a rolling upgrade. It's not a replacement for proper or a solution for permanently failed nodes. If a node is gone for good, you need a different mechanism (like anti-entropy repair) to bring a replacement up to speed.
In a typical Dynamo-style system, a write to key K is replicated to N nodes (say N=3). The coordinator identifies the 3 responsible nodes from the consistent hash ring.
Normal path: Write to nodes A, B, C. All acknowledge. Done.
Node B is down: Write to nodes A and C normally. For B's copy, write to node D instead, with a hint: "this belongs to B." D stores it in a separate "hints" table.
B recovers: D detects that B is back (via gossip), reads the hints table, forwards the data to B, and deletes the hint locally.
| Phase | What Happens | Where Data Lives |
|---|---|---|
| Normal write | A, B, C all receive the write | A, B, C |
| B goes down | Write goes to A, C, and D (with hint for B) | A, C, D (hint) |
| B recovers | D forwards hint to B | A, B, C (D deletes hint) |
Sloppy connection: Hinted handoff enables "sloppy quorums", writes succeed as long as W nodes acknowledge, even if some of those W nodes aren't the "correct" replicas. This increases write availability at the cost of slightly weaker consistency guarantees.
Hint expiration: Hints don't live forever. If B doesn't recover within a configurable window (typically hours to days), the hints are discarded. At that point, you rely on anti-entropy (read repair, Merkle tree comparison) to fix the missing data.