We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Consistency Models
When a replica is temporarily down, another node stores writes intended for it as 'hints.' Once the replica recovers, the hints are replayed to catch it up.
In short
Hinted handoff is a failure-handling technique in distributed databases where, if a replica that is supposed to receive a write is temporarily down, another node accepts the write and stores it as a "hint" (the data plus a note saying which node it really belongs to). When the down replica comes back, the holding node replays those hints to it so it catches up on everything it missed.
In short
Hinted handoff is a failure-handling technique in distributed databases where, if a replica that is supposed to receive a write is temporarily down, another node accepts the write and stores it as a "hint" (the data plus a note saying which node it really belongs to). When the down replica comes back, the holding node replays those hints to it so it catches up on everything it missed.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Hinted Handoff across a run of lessons, not one page. These 4 alone are about 189 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
Store writes temporarily on behalf of unavailable nodes. Dynamo's approach to write availability.
What is quorum in system design? How read and write quorums work, why R + W > N, sloppy quorums, hinted handoff and quorum vs consensus, with a lab.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
Scaling reads by replicating data to follower databases, primary-replica architecture and replication lag.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Hinted Handoffpay once,
yours for life
In a replicated database, every write is supposed to land on several copies (replicas) so the data survives a node failure. But nodes go down all the time for reboots, network blips, or hardware faults. The question is: what happens to a write that targets a replica that is offline right now?
One bad answer is to reject the write and tell the client to wait until the replica is back. That hurts availability. Hinted handoff takes a different path. Instead of failing, a healthy node steps in, accepts the write on behalf of the dead replica, and stashes it locally as a hint. A hint is just the write payload plus metadata recording which node was the real intended target.
The holding node keeps these hints in a separate place from its own normal data. It is not pretending the data belongs to it. It is babysitting the data temporarily so the cluster can keep accepting writes during the outage.
A coordinator node receives a write and computes which replicas should hold it, usually using consistent hashing to map the key onto the ring of nodes. Say replicas A, B, and C should each store the value but B is unreachable.
The coordinator sends the write to A and C as normal, and for B it picks a live node to hold a hint. In some designs the coordinator itself stores the hint; in others, like Amazon Dynamo, a different healthy node in the preference list stores it. The hint is tagged with B's identity and a timestamp.
Each node runs a background process that periodically checks whether the nodes it holds hints for are alive again. When B recovers, the holding node streams the queued hints to B, B applies them, and once B confirms receipt the holding node deletes its copy. Hints are usually limited in time (by default Cassandra stops storing new hints for a node that has been down longer than 3 hours) so an outage that drags on does not let hints pile up forever and eat disk.
Hinted handoff alone does not guarantee the replica catches up perfectly. Hints can be dropped after their TTL, or the holding node itself can crash and lose them. That is why systems pair it with a slower, exhaustive repair mechanism such as anti-entropy with Merkle trees and read repair to reconcile any divergence the hints missed.
Hinted handoff is the default in leaderless, eventually consistent systems that prize write availability, like Apache Cassandra and Riak. If your priority is that writes almost never fail even when nodes flap, this is a natural fit. It keeps the cluster taking traffic through short outages without forcing clients to retry.
The cost is that it weakens consistency. While a replica is down, you might satisfy a write quorum partly with hints that have not yet reached their true home. A read right after recovery, before hints are replayed, can return stale data. This is why hinted handoff is not counted toward the consistency level in Cassandra by default: a write to two real replicas plus one hint does not count as three durable copies.
It also adds load right when a cluster is fragile. When a recovered node comes back, it gets hit with a burst of replayed hints on top of normal traffic, which can slow it further. For systems that need strong consistency, such as a financial ledger using synchronous replication or consensus through Raft or Paxos, hinted handoff is the wrong tool. There you want the write to block or fail rather than be papered over with a temporary hint.
Apache Cassandra is the clearest case. Suppose a key hashes to three nodes and you write at consistency level QUORUM, meaning two of three must acknowledge. One of the three is rebooting for a kernel patch. The coordinator gets acks from the two live replicas, satisfies the quorum, and returns success to the client. For the down node it stores a hint in its local hints files.
Ten minutes later the patched node rejoins the ring. A live node notices, opens a stream, and replays every hint it accumulated during those ten minutes. The recovered node applies the writes and is now consistent with its peers for those keys. If the outage had exceeded the configured max_hint_window_in_ms, the coordinators would have stopped storing new hints for it once the window passed, and Cassandra would rely on read repair and anti-entropy repair to fix the gap instead.
Amazon's original Dynamo paper introduced this idea under the same name, and it influenced DynamoDB, Riak, and Voldemort. The shared goal across all of them is the same: keep accepting writes during partial failures, then quietly heal the laggard replicas once they return.
Apache Cassandra
Stores hints in a local table when a replica is down and replays them on recovery, with a default 3 hour hint window before falling back to repair.
Amazon Dynamo
The internal storage system whose 2007 paper coined hinted handoff, having a healthy node in the preference list hold writes for a temporarily failed replica.
Riak
Uses hinted handoff so writes succeed during node outages, then transfers stored hints back to the owning vnode once it returns.
Voldemort
LinkedIn's distributed key-value store adopted Dynamo-style hinted handoff to preserve write availability through transient failures.
No. A quorum is how many replicas must acknowledge a write for it to count as successful. Hinted handoff is what happens to the writes meant for replicas that are down. In Cassandra a hint does not count toward the quorum by default, so a write can succeed at quorum using only the live replicas while a third node's copy waits as a hint.
The hints can be lost, because they live only on that one holding node, not replicated themselves. This is why hinted handoff is a best-effort optimization, not a durability guarantee. Systems back it up with read repair and anti-entropy repair using Merkle trees to find and fix any data that the lost hints would have carried.
No, it is an availability and eventual-consistency mechanism. While hints are pending, different replicas hold different data, so a read can return stale results until the hints are replayed. For strong consistency you need synchronous replication or a consensus protocol like Raft or Paxos, not hinted handoff.
If a node stays down longer than the hint window, replaying a huge backlog would overload it on recovery and the hints would consume large amounts of disk on the holding node. Cassandra defaults to a 3 hour window, after which it drops the hints and relies on full repair to reconcile the long-absent node.
Eventual Consistency
A consistency model where updates propagate asynchronously and all replicas will eventually converge to the same value. Trades immediacy for availability.
Quorum
The minimum number of nodes that must agree for a read or write to succeed. With N replicas, W+R > N ensures overlap between write and read sets for consistency.
Read Repair
A technique where a read operation detects stale data on a replica and triggers a background update to bring it in sync. Used by Cassandra and DynamoDB to heal inconsistencies lazily.
Strong Consistency
A guarantee that after a write completes, all subsequent reads will return the updated value. Safer but slower than eventual consistency.
Linearizability
The strongest consistency guarantee: every operation appears to take effect atomically at some point between its invocation and completion. As if there's a single copy of the data.
Serializability
A guarantee that concurrent transactions produce the same result as if they were executed one at a time in some serial order. The gold standard for database transaction isolation.
This page explains the idea. The "Hinted Handoff" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Hinted Handoff lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Hinted Handoff in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
