Hinted Handoff
When a replica is temporarily down, another node stores writes intended for it as 'hints.' Once the replica recovers, the hints are replayed to catch it up.
What is Hinted Handoff?
In short
Hinted handoff is a failure-handling technique in distributed databases where, if a replica that is supposed to receive a write is temporarily down, another node accepts the write and stores it as a "hint" (the data plus a note saying which node it really belongs to). When the down replica comes back, the holding node replays those hints to it so it catches up on everything it missed.
What It Is
In a replicated database, every write is supposed to land on several copies (replicas) so the data survives a node failure. But nodes go down all the time for reboots, network blips, or hardware faults. The question is: what happens to a write that targets a replica that is offline right now?
One bad answer is to reject the write and tell the client to wait until the replica is back. That hurts availability. Hinted handoff takes a different path. Instead of failing, a healthy node steps in, accepts the write on behalf of the dead replica, and stashes it locally as a hint. A hint is just the write payload plus metadata recording which node was the real intended target.
The holding node keeps these hints in a separate place from its own normal data. It is not pretending the data belongs to it. It is babysitting the data temporarily so the cluster can keep accepting writes during the outage.
How It Works Under The Hood
A coordinator node receives a write and computes which replicas should hold it, usually using consistent hashing to map the key onto the ring of nodes. Say replicas A, B, and C should each store the value but B is unreachable.
The coordinator sends the write to A and C as normal, and for B it picks a live node to hold a hint. In some designs the coordinator itself stores the hint; in others, like Amazon Dynamo, a different healthy node in the preference list stores it. The hint is tagged with B's identity and a timestamp.
Each node runs a background process that periodically checks whether the nodes it holds hints for are alive again. When B recovers, the holding node streams the queued hints to B, B applies them, and once B confirms receipt the holding node deletes its copy. Hints usually have a time-to-live (Cassandra defaults to 3 hours) so an outage that drags on does not let hints pile up forever and eat disk.
Hinted handoff alone does not guarantee the replica catches up perfectly. Hints can be dropped after their TTL, or the holding node itself can crash and lose them. That is why systems pair it with a slower, exhaustive repair mechanism such as anti-entropy with Merkle trees and read repair to reconcile any divergence the hints missed.
When To Use It And The Trade-offs
Hinted handoff is the default in leaderless, eventually consistent systems that prize write availability, like Apache Cassandra and Riak. If your priority is that writes almost never fail even when nodes flap, this is a natural fit. It keeps the cluster taking traffic through short outages without forcing clients to retry.
The cost is that it weakens consistency. While a replica is down, you might satisfy a write quorum partly with hints that have not yet reached their true home. A read right after recovery, before hints are replayed, can return stale data. This is why hinted handoff is not counted toward the consistency level in Cassandra by default: a write to two real replicas plus one hint does not count as three durable copies.
It also adds load right when a cluster is fragile. When a recovered node comes back, it gets hit with a burst of replayed hints on top of normal traffic, which can slow it further. For systems that need strong consistency, such as a financial ledger using synchronous replication or consensus through Raft or Paxos, hinted handoff is the wrong tool. There you want the write to block or fail rather than be papered over with a temporary hint.
A Concrete Example
Apache Cassandra is the clearest case. Suppose a key hashes to three nodes and you write at consistency level QUORUM, meaning two of three must acknowledge. One of the three is rebooting for a kernel patch. The coordinator gets acks from the two live replicas, satisfies the quorum, and returns success to the client. For the down node it stores a hint in the system.hints table.
Ten minutes later the patched node rejoins the ring. A live node notices, opens a stream, and replays every hint it accumulated during those ten minutes. The recovered node applies the writes and is now consistent with its peers for those keys. If the outage had exceeded the configured max_hint_window_in_ms, those hints would have been discarded, and Cassandra would rely on read repair and anti-entropy repair to fix the gap instead.
Amazon's original Dynamo paper introduced this idea under the same name, and it influenced DynamoDB, Riak, and Voldemort. The shared goal across all of them is the same: keep accepting writes during partial failures, then quietly heal the laggard replicas once they return.
Where it is used in production
Apache Cassandra
Stores hints in a local table when a replica is down and replays them on recovery, with a default 3 hour hint window before falling back to repair.
Amazon Dynamo
The internal storage system whose 2007 paper coined hinted handoff, having a healthy node in the preference list hold writes for a temporarily failed replica.
Riak
Uses hinted handoff so writes succeed during node outages, then transfers stored hints back to the owning vnode once it returns.
Voldemort
LinkedIn's distributed key-value store adopted Dynamo-style hinted handoff to preserve write availability through transient failures.
Frequently asked questions
- Is hinted handoff the same as a write quorum?
- No. A quorum is how many replicas must acknowledge a write for it to count as successful. Hinted handoff is what happens to the writes meant for replicas that are down. In Cassandra a hint does not count toward the quorum by default, so a write can succeed at quorum using only the live replicas while a third node's copy waits as a hint.
- What happens if the node holding the hints also crashes?
- The hints can be lost, because they live only on that one holding node, not replicated themselves. This is why hinted handoff is a best-effort optimization, not a durability guarantee. Systems back it up with read repair and anti-entropy repair using Merkle trees to find and fix any data that the lost hints would have carried.
- Does hinted handoff make a system strongly consistent?
- No, it is an availability and eventual-consistency mechanism. While hints are pending, different replicas hold different data, so a read can return stale results until the hints are replayed. For strong consistency you need synchronous replication or a consensus protocol like Raft or Paxos, not hinted handoff.
- Why do hints have a time limit?
- If a node stays down longer than the hint window, replaying a huge backlog would overload it on recovery and the hints would consume large amounts of disk on the holding node. Cassandra defaults to a 3 hour window, after which it drops the hints and relies on full repair to reconcile the long-absent node.
Learn Hinted Handoff hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Hinted Handoff as part of a larger topic.
See also
Related glossary terms you might want to look up next.
Eventual Consistency
A consistency model where updates propagate asynchronously and all replicas will eventually converge to the same value. Trades immediacy for availability.
Quorum
The minimum number of nodes that must agree for a read or write to succeed. With N replicas, W+R > N ensures overlap between write and read sets for consistency.
Read Repair
A technique where a read operation detects stale data on a replica and triggers a background update to bring it in sync. Used by Cassandra and DynamoDB to heal inconsistencies lazily.
Strong Consistency
A guarantee that after a write completes, all subsequent reads will return the updated value. Safer but slower than eventual consistency.
Linearizability
The strongest consistency guarantee: every operation appears to take effect atomically at some point between its invocation and completion. As if there's a single copy of the data.
Serializability
A guarantee that concurrent transactions produce the same result as if they were executed one at a time in some serial order. The gold standard for database transaction isolation.