Distributed Lock

On a single machine, when two threads want to touch the same data, you use a mutex. The operating system guarantees only one thread holds it. A distributed lock solves the same problem, but the competing parties are separate processes running on separate servers that share no memory and only talk over a network that can drop, delay, or reorder messages.

The point is mutual exclusion across the fleet. Say five copies of your service are all running a scheduled job that charges a customer. Without coordination, all five could fire at once and charge the card five times. A distributed lock makes four of them wait or skip while one does the work.

The key fact that makes this hard: the thing holding the lock can die at any moment, and the network can lie. So a real distributed lock is not just a flag in shared storage. It needs an expiry (a lease) so a crashed holder does not block everyone forever, and it needs a way to safely hand the lock to someone else.

The simplest correct pattern uses an atomic set-if-not-exists with a timeout. In Redis this is SET resource_key random_token NX PX 30000, which means create the key only if it does not exist and auto-delete it after 30 seconds. The random token is the owner's proof of ownership. To release, the owner runs a small Lua script that deletes the key only if the stored token still matches its own. That match-then-delete must be atomic, otherwise you can delete a lock that already expired and was grabbed by someone else.

ZooKeeper and etcd take a different approach built on consensus. A client creates an ephemeral node, and ZooKeeper deletes that node automatically the moment the client's session disconnects. Clients line up in order and watch the node ahead of them, so the lock is handed off cleanly without polling. Because these systems run a quorum (typically Raft in etcd, Zab in ZooKeeper), they survive node failures and give a consistent view of who holds the lock.

The subtle failure is the paused holder. A process can acquire the lock, then get frozen by a long garbage collection pause or a network partition. Its lease expires, someone else takes the lock, and then the original process wakes up still believing it holds the lock and writes to the resource. The defense is a fencing token: a number that only ever increases, handed out with each lock grant. The protected resource rejects any write carrying a token lower than the highest one it has seen, so the stale holder's write is refused.

Reach for a distributed lock when correctness depends on exactly one actor doing something at a time across machines: a leader running a cron job once, exclusive access to a non-transactional external API, or a migration that must not run twice. It is the wrong tool for high-throughput contention. If thousands of requests fight over one lock, you have serialized your whole system through a single chokepoint, and latency and timeouts pile up.

Before adding a lock, check whether the database can do the job. A unique constraint, an UPDATE ... WHERE version = ? optimistic check, or SELECT FOR UPDATE inside a transaction often removes the need for an external lock entirely and is harder to get wrong. Idempotency keys, where repeating an operation is simply a no-op, can sidestep locking altogether.

The trade-offs are real. Single-Redis locks are fast but lose correctness if that Redis fails over mid-lock. The Redlock algorithm spreads the lock across several independent Redis nodes for better availability, but Martin Kleppmann's well-known critique showed it still does not guarantee safety under clock drift and GC pauses without fencing tokens. ZooKeeper and etcd give stronger guarantees because they are consensus systems, at the cost of more latency and operational weight.

Picture an e-commerce platform with a nightly job that reconciles inventory against a supplier's API. The service runs on eight pods in Kubernetes. The supplier's API is not idempotent, so running the reconciliation twice corrupts stock counts.

Each pod tries to acquire a lock in etcd on the key reconcile/2026-06-14 with a 10 minute lease. etcd's consensus guarantees exactly one pod wins. That pod renews its lease in the background while it works, so the lock never expires mid-run. The other seven see the key already held and exit immediately.

etcd hands the winner a fencing token tied to the lease revision. Every write the pod makes to the internal inventory store includes that token, and the store rejects stale tokens. So even if the winning pod stalls on a long GC pause, its lease lapses, another pod takes over with a higher token, and any late write from the frozen pod is rejected rather than silently overwriting fresh data.