System design interview guide
Google System Design Interview: Design Zanzibar, Google's Permission System
Short answer
A strong scenario to practice for a Google system design interview is Zanzibar, the service that, in Google's 2019 paper, decided who may see or edit a Drive file, a photo or a video. We have no public source that Google asks it. The key idea is one shared service that stores permissions as small records, such as "user 10 is an owner of doc:readme", plus rules like "every editor is a viewer". Checks must respect the order of changes, so a user removed from a document never sees content added later. Zanzibar did this with zookies, tokens holding Spanner timestamps. The 2019 paper reported more than 2 trillion records and more than 95% of checks within 10 milliseconds.
Asked to design the file storage and sync behind Drive instead of its permission checks? Read the Google Drive system design guide
Two courses by the author of this page:
770 lessons · 18 free to read
₹1,111 in India · $35 elsewhere, once
Get System DesignYou own this course
219 lessons · 10 free to read
₹1,333 in India · $35 elsewhere, once
Get Generative AI and Agentic AIYou own this course
Save your progress and get the interview checklist PDF
Every time you open a shared file in Google Drive, something must first answer one question: is this person allowed to see it? In its 2019 paper, Google wrote that this answer came from Zanzibar. The paper said Zanzibar stored more than 2 trillion permission records and served more than 10 million queries a second. It answered more than 95% of permission checks within 10 milliseconds. Calendar, Cloud, Drive, Maps, Photos and YouTube all used it then.
Designing Zanzibar means building one service that answers "can user U do R to object O?" for many Google products. Google's 2019 paper describes Zanzibar as its global authorization system, the part that decided who may see or edit a document, photo or video. Each permission is a relation tuple, a record such as "user 10 is an owner of doc:readme". Each product writes rules like "every editor is a viewer". The hard part is the new enemy problem: someone you removed must not see content added after the removal. Zanzibar used zookies, tokens holding timestamps from Spanner, Google's global database. Caching, an index called Leopard and hedged requests (a second copy of a slow request) kept it fast.
Where it shows up
Authorization, deciding who may do what, is a common system design topic for backend, infrastructure and security roles. It also appears as design an access control system, design Google Drive sharing or design a permission service. We have no public source that Google uses this exact question in its interviews. This guide is built from Google's own 2019 paper on Zanzibar, so you can reason with real numbers.
Spec sheetGoogle
- 01Size of one tuple, estimated
- ~50 bytes
- 02Reads compared with writes
- ~500 to 1
- 03Chance one search page hits a slow check, estimated
- ~63%
- 04Cache lookups compared with Spanner reads
- ~10 to 1
worked through below, with the maths
Google in its own numbers
Every figure below is quoted from Google's own engineering writing. Tap a source to check it.
2T+
relation tuples, close to 100 TB, in 2019
Zanzibar paper, USENIX ATC 2019, section 410M+
client queries a second, in 2019
Zanzibar paper, USENIX ATC 2019, section 44.2M
Check requests a second at peak, December 2018
Zanzibar paper, USENIX ATC 2019, section 495% < 10 ms
of authorization checks answered, in 2019
Zanzibar paper, USENIX ATC 201999.999%+
availability over the 3 years before 2019, counting answers within 5 s (Safe) or 15 s (Recent)
Zanzibar paper, USENIX ATC 201910,000+
servers in several dozen clusters, in 2019
Zanzibar paper, USENIX ATC 2019, section 4
Why this question is asked
This question tests whether you can keep a security decision correct while it is spread over the whole world. A permission check is small, but it sits in front of almost every request. It must be fast, it must almost never be down, and it must never let the wrong person in. Those goals pull against each other. Reading the newest data everywhere is slow; reading a local copy can be out of date. The interviewer wants to hear how you store permissions, how you follow groups inside groups, and how you stop an old copy from leaking new content. A candidate who says "cache the ACL (access control list, the list of who may do what) in Redis" and stops has missed the point.
What the interviewer asks
“Design a permission service that many products share, like Google's Zanzibar. Products such as Drive, Photos and YouTube call it to ask whether a user may view or edit an object. It must answer in a few milliseconds, work from many regions, and respect the order of changes. If a user is removed from a folder, they must not see anything added after that. Say this first: "I will store every permission as a small record, user, relation, object, and let each product write rules such as every editor is also a viewer. Checks will read a nearby copy for speed, and I will use a consistency token saved with each content version, so a removed user never sees content added after the removal." Then spend the time like this: requirements and numbers (5 minutes), the record model and rules (10), the check path and caching (10), the ordering problem and the token (10), then hot spots, slow requests and failure (10).”
Ask these back before you draw
- 1.Which products call us, and do they all share one data model? (Assume many products, one model, as in the 2019 paper.)
- 2.What kinds of rules? Direct sharing, groups inside groups, and inheriting from a parent folder? (Yes, all three.)
- 3.How fast must a check be? (The paper's 2019 figure: more than 95% of checks within 10 ms.)
- 4.How fresh must the answer be? (Assume a few seconds out of date is fine, except that new content must never be shown to someone removed before it was added.)
- 5.Do we also need to list who has access, for example to build a search index? (Zanzibar has an Expand call for this.)
This page is the free part.
The course goes deeper on the Google design
₹1,111 in India$35 everywhere elseonce, for the whole course
The System Design course covers the Google design across a run of lessons, not one page. These 4 alone are about 65 minutes of step-by-step reading, every one with a quiz.
- Authorization
Controlling what authenticated users are allowed to do: permissions, roles, and access policies.
- Role-Based Access Control (RBAC)
Assigning permissions to roles instead of individuals, the most widely used access control model in production systems.
- Attribute-Based Access Control (ABAC)
Making access decisions based on attributes of users, resources, and environment, the most flexible access control model.
- Policy-Based Access Control
Defining access rules as declarative policies evaluated by a central decision engine, OPA, Cedar, and beyond.
all part of
System Design Masterclass
770 lessons · about 290 hours · 18 free to read
₹1,111in India, by UPI
$35everywhere else, by card or PayPal
It is really nice and good Please join the course
You own this course
Continue with Authorizationpay once,
yours for life
Live from the course bench
Before you design Google, try the questions it rests on.
Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.
01Requirements
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Check: answer whether user U has relation R (such as viewer or editor) to object O
- Write: add or remove a permission, for one tuple or for all of an object's tuples safely
- Read: list the stored permissions of an object or the members of a group
- Expand: return the tree of users and groups that hold a relation, following the config's rules, so a client can build a search index that respects permissions
- Watch: stream every permission change in order, so other systems can keep their own copies
- Let each product define its own relations and rules in a namespace config
- Support groups inside groups, and inheriting access from a parent folder
Non-functional requirements
- Fast: in 2019 the paper reported more than 95% of checks within 10 milliseconds
- Highly available: the paper reported more than 99.999% over 3 years, where a request counted as answered if it came back within 5 seconds, or 15 seconds for the freshest reads. A product that cannot check must deny access, so an outage locks users out
- Correct under change: respect the order of permission and content changes (the new enemy problem)
- Global: data close to users in every region, because a check for any object can come from anywhere
- Huge and read-heavy: trillions of tuples, and reads far more common than writes
- Isolated: one product's spike of traffic must not slow every other product down
02Scale estimates
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Size of one tuple, estimated
~50 bytes
The paper gives close to 100 terabytes for more than 2 trillion relation tuples in 2019. 100 TB / 2 trillion is about 50 bytes a tuple. That is our arithmetic, not Google's number. It tells you a tuple is a short key, not a document. If close to 100 TB is one copy, more than 30 full copies put the total above 3 PB. The paper does not say which, so treat it as a range.
Reads compared with writes
~500 to 1
December 2018 peaks from the paper: Check about 4.2M a second, Read 8.2M, Expand 760K, Write 25K. Adding the three read peaks, which may not fall at the same moment, gives at most about 13.2M a second. 13.2M / 25K is about 500. The paper puts it as two orders of magnitude. Design the read path first.
Chance one search page hits a slow check, estimated
~63%
The paper says a search can need tens to hundreds of checks. Assume 100 checks run in parallel, and each one has a 1 in 100 chance of being slower than its 99th percentile. Assume they are independent. Then 1 - 0.99^100 is about 0.63. So most pages wait for at least one slow check. This is why tail latency matters and why Zanzibar hedges requests.
Cache lookups compared with Spanner reads
~10 to 1
The paper's 2019 peak: about 200 million in-memory cache lookups a second, against only about 20 million read calls a second to Spanner. 200M / 20M is 10. Most lookups miss: the hit rates were 2% to 24%, so this is not 10 hits for each read. The paper credits caching together with pooling of reads for keeping Spanner traffic that low.
Where a Safe request begins
10 s
The paper says storage replication heartbeats ran every 8 seconds. It calls a request Safe when its zookie is more than 10 seconds old. Safe requests could usually be served inside the region, and were about 100 times as common as Recent ones. So most checks can read data a few seconds old without breaking any promise.
03Architecture
High-level architecture
Start with a product such as Drive. When a user opens a document, Drive calls Zanzibar's Check API before it shows anything. Zanzibar's main servers are called aclservers (ACL means access control list, the list of who may do what). In 2019 they ran in clusters around the world. A request can arrive at any aclserver in a cluster. That server splits the work and sends parts to other aclservers, which may send parts on again, for example to walk a chain of nested groups. The first server gathers the answer and returns yes or no. Underneath, every tuple lives in Spanner, Google's globally distributed database. The paper describes one Spanner database of tuples for each client namespace, one for all namespace configs, and one changelog shared by all namespaces. Watchservers read the changelog and stream changes to clients that keep their own copies. A separate index called Leopard handles groups that are deeply nested or have very many child groups. An offline pipeline makes regular snapshots of the tuples and removes very old versions. In 2019 the data was copied to more than 30 locations, so most checks read a nearby copy.
In a real interview, sketch this on the whiteboard before diving into any single box.

04One request
What happens when someone opens a shared Google Doc, step by step
This walk follows the 2019 Zanzibar paper. Which product calls Zanzibar, and where it stores the zookie, is up to the product; the Drive example and the order of the first step are this design's.
- 1
The product receives the request
Bob opens doc:readme in Drive. In this design, Drive loads the document and the zookie it saved with the current version of its content. A zookie is a small token that carries a timestamp.
- 2
Drive asks Zanzibar
Drive sends a Check request: does Bob have the viewer relation to doc:readme? It sends the zookie too. The zookie says how fresh Zanzibar's data must be: at least as new as that content version.
- 3
Zanzibar picks a snapshot
The aclserver picks one point in time to read the data at, never older than the zookie. It rounds the time up to a coarse step, such as one or ten seconds, so many checks share one snapshot and its cached results.
- 4
The rules are expanded
The namespace config for doc says a viewer is anyone stored as a viewer, plus every editor, plus every viewer of the parent folder. Zanzibar turns this into a small tree of yes or no questions.
- 5
The questions run in parallel
Zanzibar evaluates the leaves of that tree at the same time, often on other aclservers chosen by the object ID. It reads tuples from a nearby Spanner copy, or asks Leopard for a large nested group. As soon as one branch says yes, the rest are cancelled.
- 6
The answer comes back
Zanzibar returns yes or no. Drive shows the document, or an access denied page. In December 2018, the median Safe check (one whose zookie was more than 10 seconds old) took about 3 ms.
05Components
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Relation tuples
The basic record. The paper writes it as object#relation@user, for example doc:readme#owner@10, meaning user 10 is an owner of doc:readme. The user part can itself be a set, like group:eng#member. That is how a document can be shared with a whole group, and how groups can contain groups.
Namespace configs
Each product configures a namespace, such as doc or folder, before storing tuples in it. The config names the relations (owner, editor, viewer) and the rewrite rules between them. In 2019 Zanzibar had more than 1,500 namespaces from hundreds of client applications. Config files ran from tens of lines to thousands, with the median near 500.
aclservers
The main servers. They answer Check, Read, Expand and Write. They fan work out to each other and cache results. In 2019 the paper counted more than 10,000 Zanzibar servers in several dozen clusters, with the median cluster near 500 servers.
Spanner
Google's globally distributed database, described in a 2012 paper. Zanzibar stores tuples, configs and the changelog in it. Spanner gives every write a timestamp from TrueTime, a clock service with a known error bound. Those timestamps follow the real order of writes. The paper notes that Zanzibar ran on Google's internal Spanner, not on Cloud Spanner.
Changelog and watchservers
Every Zanzibar write goes into the tuple table and a changelog shard in one transaction. Watchservers read the changelog and stream changes in timestamp order. Each response ends with a heartbeat zookie, a timestamp up to which the client has every change. The client resumes from it, so nothing is skipped across shards. Clients use this to keep their own indexes up to date.
Leopard index
A separate index for groups that are deeply nested or have very many child groups. It stores flattened sets of integers so that "is user U in group G?" becomes one intersection of two sorted lists. An offline builder makes it from snapshots, and an online layer applies changes from the Watch stream on top.
Distributed cache and lock table
The aclservers in a cluster share a cache of reads and check results, spread with consistent hashing. A lock table makes sure that when many identical requests arrive together, only one does the work. The others wait for its result.
06Data model
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
relation_tuples (one Spanner database per namespace)shard_idobject_idrelationuser (a user ID or a userset)commit_timestampThis primary key is the one the 2019 paper gives. Each version of a tuple is a separate row, so Zanzibar can answer at any timestamp inside the garbage collection window, the period before old versions are deleted. The shard ID decides which slice of the database holds the row. Usually it comes only from the object ID. For a group with a huge number of members, it is computed from the object ID and the user.
changelog (one database shared by all namespaces)changelog_shard_idtimestampunique_update_idFrom the paper: a changelog shard is picked at random for each write, and the write commits to the tuple table and the changelog in one transaction. The Watch API reads from here.
namespace_configsnamespace_idconfigconfig_changelog (keyed by commit timestamp)From the paper: one table holds the configs, keyed by namespace ID. A second table is a changelog of config changes. Servers load every config at startup and then follow the changelog. All servers working on one request use the same config snapshot.
documents (in the product, not in Zanzibar)doc_idcontent_versioncontentzookieThe paper says the client stores the zookie together with the content change, in one atomic write to its own storage. The table and field names here are this design's. The point is that the zookie travels with the content, so a later check knows how fresh its data must be.
07Deep dives
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Relation tuples and the namespace config
Zanzibar stores permissions as tuples, not as one list per object. A tuple says that an object has a relation to a user. The paper's examples: doc:readme#owner@10 means user 10 is an owner of doc:readme. group:eng#member@11 means user 11 is a member of group:eng. doc:readme#viewer@group:eng#member means every member of group:eng can view doc:readme. doc:readme#parent@folder:A#... means the document sits in folder A. Storing a tuple for every implied permission would be wasteful. So each namespace config holds rewrite rules instead. The paper's example says every owner is an editor, every editor is a viewer, and every viewer of the parent folder is a viewer of the document. There are three building blocks. _this means the tuples stored directly. computed_userset points to another relation on the same object. tuple_to_userset follows a relation such as parent to another object and reads a relation there. Rules can be combined with union, intersection and exclusion. The paper says roles in role-based access control (RBAC) are similar to these relations, and that several products built RBAC on top.
name: "doc"
relation { name: "owner" }
relation { name: "editor"
userset_rewrite { union {
child { _this {} }
child { computed_userset { relation: "owner" } } } } }
relation { name: "viewer"
userset_rewrite { union {
child { _this {} }
child { computed_userset { relation: "editor" } }
child { tuple_to_userset {
tupleset { relation: "parent" }
computed_userset { object: $TUPLE_USERSET_OBJECT relation: "viewer" } } } } } }The new enemy problem, and why zookies fix it
The paper names the main correctness risk the new enemy problem. Its first example: Alice removes Bob from a folder. Then she asks Charlie to move new documents into the folder. If the check ignores the order of those two changes, Bob may see the new documents. Its second example: Alice removes Bob from a document, then adds new content. If the check uses an old copy of the ACL from before the removal, Bob may see the new content. Always reading the newest data everywhere would fix this, but it needs slow trips across the world. Zanzibar uses zookies instead. When a user saves new content, the product sends a content-change check. That check runs on the newest data. If it passes, Zanzibar returns a zookie holding a timestamp later than every earlier ACL change. The product stores the zookie with the content. Later checks on that content send the zookie, and Zanzibar promises to read data at least that fresh. Because the removal was acknowledged before the save began, Spanner gave it the smaller timestamp, so any snapshot that fresh already includes the removal. In this design, the product waits for the removal to be acknowledged before it lets anyone save new content into that document or folder. Note what zookies do not promise: they protect new content only. A check on an older version may still run at a snapshot from before the removal, so the removed user can open that older version for a few seconds. The paper chose an opaque token, not a raw timestamp, so clients cannot pick arbitrary times.
Spanner and TrueTime underneath
The zookie only works if timestamps follow the real order of events. Zanzibar gets that from Spanner. The 2012 Spanner paper says Spanner gives transactions globally meaningful commit timestamps. If transaction T1 commits before T2 starts, T1 gets the smaller timestamp. This property is called external consistency. Google's Cloud Spanner docs say that under it, the system behaves as if all transactions run one after another. The tool behind it is TrueTime. It returns the time as an interval with a known error, kept small by GPS and atomic clocks. In 2012 the paper reported that the error usually moved between about 1 and 7 milliseconds. Spanner then waits out that error before it makes a write visible; this is called commit wait. Zanzibar reads every check at a single snapshot timestamp, so it sees all writes up to that time and none after. The 2019 paper also describes the copies. Zanzibar had 5 voting replicas, in the eastern and central US, in 3 different metro areas. They were within 25 milliseconds of each other so that Paxos, the voting protocol that commits writes, could finish quickly. Every write needs this agreement among Spanner servers, which is why the paper reports writes as the slowest call, about 127 ms at the median in December 2018.
Evaluating a check, and the Leopard index
A check becomes a tree of yes or no questions. Is there a tuple saying Bob is a viewer? Is Bob an editor? Is Bob a viewer of the parent folder? Each of those can lead to more, because a group can contain groups. The paper calls this pointer chasing. To keep it fast, Zanzibar evaluates all the leaves of the tree at the same time. When one result settles a branch, it cancels the rest of that branch. It also pools reads for the same check, so it sends fewer calls to Spanner. Pointer chasing is slow for groups that are very deep or very wide. For those namespaces Zanzibar uses Leopard. Leopard flattens the graph into two kinds of sets. One maps a group to every group nested inside it. The other maps a user to the groups they belong to directly. Then "is Bob in group G?" is one question: do those two sets overlap? The sets are sorted lists of integers, so an intersection is fast. In this design, a walk without Leopard keeps a set of groups already visited and a depth limit, because groups can end up nested inside each other in a loop. Leopard is rebuilt offline from a snapshot. On top of it, an online layer stores every change since that snapshot with its timestamp, fed by the Watch stream. A query merges in only the changes up to the query's timestamp, so a check that uses Leopard still reads at one consistent snapshot, at least as fresh as the zookie. One tuple change can potentially create tens of thousands of Leopard updates. In 2019 Leopard answered in under 150 microseconds at the median.
async def check(user, obj, rel, snap):
rule = config(obj.namespace, rel, snap) # e.g. viewer = this | editor | parent#viewer
# union only: intersection needs every branch, exclusion needs base yes and excluded no
tasks = [asyncio.create_task(leaf(user, obj, b, snap)) for b in rule.branches]
for done in asyncio.as_completed(tasks):
if await done: # one branch said yes
for t in tasks: t.cancel() # stop the others
return True
return FalseHot spots: caching on normalized data
Zanzibar keeps its data normalized: each fact is stored once, as a tuple, and not copied into every object that depends on it. That keeps it consistent, but it makes hot spots. A search page may check many documents that all share one popular group. The paper calls handling hot spots the most critical frontier for latency and availability. Its answers stack up. The aclservers share a distributed cache, spread with consistent hashing. For most namespaces, requests are forwarded by object ID, so checks on one object land on one server and share its cache. Results are cached at both the caller and the server it calls. Cache keys include the snapshot timestamp, so a result from one snapshot is never reused for another. To make sharing likely, evaluation times are rounded up to one or ten seconds. Rounding up is safe: a Spanner read at a future time simply waits until that time has passed. A lock table stops a cache stampede, where many identical requests all miss the cache at once. For a very hot object, Zanzibar reads and caches all the tuples for that object and relation in one go. In 2019 the cache hit rates for checks looked low: 10% on the server that received a delegated check (a part of a check one aclserver hands to another), plus 12% saved by the lock table, and only 2% plus 3% on the caller's side. But that still prevented 500,000 internal calls a second from becoming hot spots.
Tail latency, hedging and isolation
Over 7 days in December 2018, the paper's Safe checks peaked at roughly 3, 11, 20 and 93 milliseconds at the 50th, 95th, 99th and 99.9th percentiles. The slow end matters most because a search page waits for its slowest check. Zanzibar uses request hedging for calls to Spanner and Leopard. It sends one request, and if that is slow, sends the same request to another server and takes whichever answers first. Each server tracks recent latency to decide when a request counts as slow, so hedging adds only a small amount of traffic. In 2019, about 1% of Spanner reads, 200,000 a second, gained from hedging. Zanzibar does not hedge between its own aclservers. Some checks are much more expensive than others, and copying the expensive ones would make latency worse. Isolation is the other half. Each client has a global limit on CPU use, counted in a hardware-neutral unit. Each server limits its outstanding calls, in total and for each client. Zanzibar also limits concurrent reads for each object and client pair, and for each client, on each Spanner server. That way one product's sudden popular feature cannot slow down the others.
Common mistakes candidates make
The first mistake is treating permissions as a column on each document. Groups inside groups and folder inheritance then turn into copies that drift apart. Say instead: store tuples, and put the inheritance rules in a config. The second mistake is caching the ACL with a time to live and stopping there. A cache that is a few seconds old is exactly what lets a removed user see new content. Say instead: tie the freshness to the content version, which is what a zookie does. The third mistake is the opposite: always reading the newest data. That needs a trip to the leader replica, the copy that orders all writes, across regions on every check. Say why most checks can use a slightly older local snapshot. The fourth mistake is ignoring hot spots. One popular group shared by many documents can overload one database server. The fifth is forgetting what happens when the service is down: products must deny access, so availability is a security property too. Spend 45 minutes like this: requirements and numbers (5), the tuple model and config (10), the check path and caching (10), the new enemy problem and zookies (10), then hot spots, tail latency and failure (10).
08Trade-offs
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
One shared authorization service vs one per product
The paper gives four reasons for one shared service: the same meaning and user experience across products, easier sharing when one product embeds another's object, shared tools such as a search index that respects permissions, and solving the hard consistency and scale problems once. The cost is that every product depends on it, so it must be faster and more available than any single product needs.
Normalized tuples vs copying permissions into every object
Copying permissions makes each check one read, but a group change must rewrite millions of objects and the copies can disagree. Zanzibar stores each fact once and relies on caching, request deduplication and the Leopard index for the slow cases. The paper allows limited denormalization only in Leopard, for deeply nested groups.
At-least-as-fresh zookies vs always reading the newest data
Reading the newest data on every check needs global synchronization, which is slow and less available. A zookie only sets a lower bound. Zanzibar can then pick any snapshot newer than it, usually one already copied to the local region. In December 2018 Safe checks were about two orders of magnitude, roughly 100 times, more common than Recent ones, and their 95th percentile averaged about 9.5 ms against 60 ms over that week. The price is that removing a user is not instant for content that already exists. A check on an older version may run at a snapshot from before the removal, so the removed user can still open that version for a few seconds. The paper's new enemy problem is only about new content. In this design, when a product must cut access at once, for example after a leak, it saves a new content version, which runs a content-change check and stores a new zookie.
Spanner with external consistency vs an eventually consistent store
The paper compares Zanzibar with Facebook's TAO, which offers eventual global consistency. Zanzibar chose Spanner so that timestamps follow the causal order, which is what defeats the new enemy problem. The price is slower writes: in December 2018 a write took about 127 ms at the median, because it needs agreement among Spanner servers. Writes were rare, so that was acceptable.
Rounding snapshot times up vs using the exact time
Exact microsecond timestamps would make almost every cache key unique. Rounding up to one or ten seconds lets most checks share a few snapshots and their cached results. Correctness is kept because a read at a later time still sees every write before it. The cost is that a rounded-up time can be newer than the data the local copy has, or even slightly in the future. Then the read waits until that time has passed, or has to fetch from another region. The paper says Google tuned the step size to match what its clients needed.
Hedge requests to storage, but not between aclservers
Hedging cuts the tail when requests cost about the same, as Spanner and Leopard reads do. Checks between aclservers vary hugely in cost, so hedging them would duplicate the most expensive work. Zanzibar instead spreads load over replicas and avoids slow servers.
Free PDF · 18 pages
Get the free System Design Interview Cheat Sheet
The interview in seven stages, the numbers worth knowing by heart, and twelve classic systems on one page each, every line linked to the lesson it comes from.
09Follow-ups
Follow-up questions to expect
After the main design, the interviewer usually picks one part and asks more. Here is what to say.
Bob was removed from a document a second ago. Can he still open the version that was already there?
Possibly, for a few seconds. A check on that version only has to be at least as fresh as the zookie saved with it, and that zookie is older than the removal. The paper's guarantee covers new content, and Bob could already see the old version. In this design, a product that must revoke at once saves a new version, which forces a check at the latest snapshot and stores a newer zookie.
Spanner is re-electing a leader, which the 2019 paper says can take up to 10 seconds. What happens to checks?
Most checks are Safe: their zookie is older than about 10 seconds, so they read data already copied to the local region and keep working. Writes, content-change checks and Recent checks need the leader or a trip to another region, so they slow down until the new leader is ready. The paper's availability measure allows 15 seconds for Recent requests for this reason. In this design, products retry within a deadline, and if the check still fails they deny access rather than guess.
Bob is removed from a document, then Alice adds new text. How do you make sure Bob cannot see it?
Saving the new text triggers a content-change check, which runs on the newest data and returns a zookie. The product stores the zookie with that version. Any later check on that version must read data at least as new as the zookie. This is the paper's second new enemy example. It works because the removal was acknowledged before the save began, so Spanner gave it the smaller timestamp. In this design, the product waits for the removal to be acknowledged before it lets the user save new content into that document or folder.
A check arrives with no zookie. Which snapshot do you use?
A default staleness, chosen so the data is almost always already copied to the local region. The paper describes servers measuring how often reads at each staleness need an out-of-zone trip (a read from another region), and lowering the default to the freshest value that stays local. It is only a speed setting; a zookie, when given, always wins.
A group has millions of members and many levels of nesting. How do you check membership fast?
Walking the nesting one level at a time is too slow. Use an index like Leopard, which flattens the group graph. Then membership is the overlap between the groups a user belongs to directly and the groups nested inside the target group. Keep the index fresh by applying the change stream on top of an offline snapshot.
Two admins edit the same document's sharing settings at once. How do you stop one from overwriting the other?
The paper describes optimistic concurrency control. Read all the object's tuples, including a special lock tuple. Then write the changes with a condition: commit only if the lock tuple has not changed since the read. If it has, read again and retry.
A search results page must check 100 documents. How do you keep it fast?
Run the checks in parallel, forward them by object ID so related checks share a cache, and deduplicate identical in-flight checks with a lock table. Hedge slow storage reads. In this design, return results as checks pass instead of waiting for all 100, if the product allows it. The paper says Drive search issued tens to hundreds of checks per page.
10In production
How Google actually does it
Every fact about Zanzibar on this page comes from Google's own paper, "Zanzibar: Google's Consistent, Global Authorization System", published at USENIX ATC in July 2019. Its traffic and latency numbers were sampled over 7 days in December 2018, and its availability covers the 3 years before publication. The paper describes the system as it was then, so quote its numbers in the past tense and say the year. The TrueTime facts are from the 2012 Spanner paper, and the definition of external consistency is from Google Cloud's Spanner docs. The paper notes Zanzibar ran on Google's internal Spanner, not Cloud Spanner, and that Google's Cloud IAM was built as a layer on top of Zanzibar. The traffic estimates, the Drive walk-through order, the documents table and anything marked "in this design" or "assume" are ours.
Sources
- Google: Zanzibar, Google's Consistent, Global Authorization System (USENIX ATC 2019)
- The Zanzibar paper as a PDF (USENIX, 2019)
- Google Research publication page for the Zanzibar paper (2019)
- Google: Spanner, Google's Globally-Distributed Database (OSDI 2012)
- Google Cloud: Spanner, TrueTime and external consistency (docs)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Authorization
intermediate / security architecture
Role-Based Access Control (RBAC)
intermediate / security architecture
Attribute-Based Access Control (ABAC)
intermediate / security architecture
Policy-Based Access Control
intermediate / security architecture
Linearizability
advanced / consistency models
Causal Consistency
advanced / consistency models
Snapshot Isolation
advanced / consistency models
Lamport Timestamps
advanced / distributed systems core
Consensus Algorithms
advanced / distributed systems core
Consistent Hashing
intermediate / data replication distribution
Cache Stampede Prevention
foundation / caching strategies
System design interviews at companies in the US, UK, Canada and Singapore
41 walkthroughs of real systems at well-known companies, each built from that company's own engineering blog, talks or filings, with the numbers cited.
Banking, payments and trading
Commerce and delivery
Rides and maps
- Grab system design (Singapore)
- Uber system design
Feeds, social and messaging
Streaming and media
AI labs and big tech
- Anthropic system design
- OpenAI system design
- ChatGPT system design
- Google system design (this page)
- Microsoft system design
Developer and collaboration tools
11FAQ
Frequently asked questions
Go deeper
This guide is built on these 6 lessons. Each one teaches one piece of the Google design above, step by step.
- AuthorizationSystem Design · security architecture · 18 min
- Role-Based Access Control (RBAC)System Design · security architecture · 18 min
- Attribute-Based Access Control (ABAC)System Design · security architecture · 11 min
- Policy-Based Access ControlSystem Design · security architecture · 18 min
- LinearizabilitySystem Design · consistency models · 25 min
- Causal ConsistencySystem Design · consistency models · 20 min
Practice with 770 system design lessons
Lifetime access for ₹1,111 in India or $35 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.
This one question is not enough for the interview.
Get all 200 questions in the System Design and Distributed Systems pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
NEWSystem Design Hands-On
Included in the System Design course: real systems, built live
Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.


