We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
6 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Distributed Systems Core
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Distributed Systems Core
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
You know that awkward dance when two people walk toward a single-stall bathroom at the same time? Someone has to go first. In the physical world, social norms (and a lock on the door) solve this. But on a computer, there's no social norm. Two threads, two processes, or two servers will happily try to access the same resource simultaneously, corrupt each other's work, and leave your data in a state that makes no sense.
On a single machine, locks are straightforward. A mutex, a semaphore, a synchronized block, the operating system handles it. But in a distributed system, there's no shared operating system. Node A in Virginia and Node B in Frankfurt can't share a mutex. They need a distributed lock, a coordination mechanism that works across network boundaries.
And this is where things get treacherous. Martin Kleppmann wrote a famous blog post in 2016 titled "How to do distributed locking" where he demolished Salvatore Sanfilippo's Redlock algorithm, the distributed lock built into . The argument exposed a fundamental tension: distributed locks are either safe but slow, or fast but unsafe. There's no free lunch.
Understanding when you need a distributed lock, which kind to use, and what guarantees it actually provides will save you from some of the worst bugs in distributed systems, the kind that only manifest under load, during network issues, and are nearly impossible to reproduce.
A lock needs three properties:
Safety (mutual exclusion), at most one client holds the lock at any given time. If two clients both think they hold the lock, the lock is broken.
Liveness (deadlock freedom), if a client that holds the lock crashes, the lock is eventually released so others can proceed. No lock should be held forever.
Fault tolerance, the lock system continues to function even if some nodes fail. A lock on a single instance disappears if that instance crashes.
The challenge in distributed systems is that these properties conflict with each other under failure scenarios:
| Scenario | Problem |
|---|---|
| Client holds lock, network partitions | Lock holder can't renew; does the lock expire? What if the holder is still working? |
| Lock service node crashes | If the lock was only on that node, it's lost. Other clients might acquire it. |
| Clock skew | Lock expires at different times on different nodes. Two clients think they have it. |
| GC pauses | Client acquires lock, JVM does a 30-second GC pause, lock expires, another client acquires it, first client resumes, both think they hold the lock. |
The GC pause scenario is particularly insidious. Your code acquires the lock, does some work, and releases it. Everything looks correct in the code. But between "acquire" and "do work," the garbage collector freezes your process for 30 seconds. The lock expires. Another process acquires it. Your process resumes, not knowing the lock expired, and proceeds to corrupt shared state.
It is easy and good
“It is really nice and good Please join the course”