May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
System Design and AI Engineering, by Roni Das. 989 interactive lessons in two courses.: 770 in System Design and 219 in AI Engineering.
Start LearningSystem design interview guide
Short answer
Kafka is a distributed commit log used as a message queue. A topic is split into partitions, and each partition is an append-only log where every record gets an offset. Producers add to the end, while consumers read forward and save their own offset, so reading deletes nothing and many consumer groups can read one topic. Each partition is copied to several brokers, a write is committed once every in-sync replica has it, and acks=all makes the producer wait for that. Order holds within a partition, not across partitions.
in this guide
7 parts open to everyone · 3 with a free account · 3 with the course
You can read 7 of 13 parts. A free account opens the deep dive; the course opens all of it.
This problem tests whether a candidate really understands the log as a primitive, not just an API they call. It forces you to reason about ordering versus parallelism, about the difference between a message being written and a message being committed, and about how replication interacts with availability during a broker failure. A strong candidate will talk about partition keys and hot partitions, about consumer group rebalancing, and about why at-least-once is the honest default and what it actually costs to get closer to exactly-once. It is a good filter because the naive answer, a queue with producers and consumers, falls apart the moment the interviewer asks how you keep order, survive a crash, and still push a million messages a second.
This page is the free part.
₹1,111 in India$35 everywhere elseonce, for the whole course
The System Design course covers the Apache Kafka design across a run of lessons, not one page. These 4 alone are about 112 minutes of step-by-step reading, 2 of them with code you run in the browser, all with a quiz.
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
This guide is built on these 6 lessons. Each one teaches one piece of the Apache Kafka design above, step by step.
Lifetime access for ₹1,111 in India or $35 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.
This one question is not enough for the interview.
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
Two courses by the author of this page:
770 lessons · 18 free to read
₹1,111 in India · $35 elsewhere, once
Get System DesignYou own this course
219 lessons · 10 free to read
₹1,333 in India · $35 elsewhere, once
Get Generative AI and Agentic AIYou own this course
Save your progress and get the interview checklist PDF
A single Kafka cluster at a large company routinely moves trillions of messages a day across thousands of topics. LinkedIn, where Kafka was born, has publicly described clusters handling on the order of 7 trillion messages per day. The trick is that a well tuned broker can sustain hundreds of megabytes per second of writes on commodity disks, because Kafka almost never does a random disk seek. These are published or widely cited figures, and exact throughput depends heavily on hardware, message size, and batching.
Kafka is a distributed, partitioned, replicated commit log dressed up as a messaging system. A topic is split into partitions, and each partition is an append-only, totally ordered sequence of records addressed by a monotonically increasing offset. Producers append to the tail, consumers read forward at their own pace and track their own offset, and the same records can be re-read by many independent consumer groups because reading does not destroy the message. Ordering is guaranteed only within a partition, which is the price you pay for horizontal scale. Durability comes from replicating each partition to several brokers, with one leader taking all reads and writes and followers pulling to stay in sync. The in-sync replica set plus the acks setting lets you dial the tradeoff between latency and how many failures you can survive without losing an acknowledged write. The whole thing is fast because it leans on sequential disk I/O, the OS page cache, and zero-copy transfer rather than clever in-memory structures. Coordination that used to live in ZooKeeper now lives inside Kafka itself through the KRaft metadata quorum.
Where it shows up
Asked at LinkedIn, Confluent, Uber, Stripe, Datadog, and most companies that run a large event backbone. It also shows up in generic interviews as "design a distributed message queue" or "design a pub-sub system," where Kafka is the reference implementation the interviewer has in mind.
Spec sheetApache Kafka
worked through below, with the maths
The hard parts of this design are taught step by step in the System Design course: Write-Ahead Logging, Leader Election and Consensus Algorithms.
Write changes to a sequential log before applying them, the foundation of database crash recovery.
How distributed nodes pick a leader without a central authority, bully algorithm, ring, and Raft elections.
Paxos, Raft, and ZAB, how distributed nodes agree on a single value despite failures.
Processing every event exactly once despite failures, the holy grail of stream processing.
all part of
System Design Masterclass
770 lessons · about 290 hours · 18 free to read
₹1,111in India, by UPI
$35everywhere else, by card or PayPal
You own this course
Continue with Write-Ahead Loggingpay once,
yours for life
Live from the course bench
Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Peak ingest
~1M messages/sec, ~500 MB/sec per cluster
Assume 1M msgs/sec at an average 500 bytes each. 1M * 500 bytes = 500 MB/sec of raw writes. Batching and compression cut the bytes on the wire and to disk well below this. Derived example, not a fixed limit.
Partitions for parallelism
~1,000 partitions for a busy topic
Consumer parallelism is capped by partition count, since one partition maps to one consumer per group. To let 1,000 consumers work in parallel you need at least 1,000 partitions. Derived from the one-partition-per-consumer rule.
Storage per day
~43 TB/day raw before replication and compression
500 MB/sec * 86,400 sec = ~43 TB/day of raw ingest. With replication factor 3 the on-disk cost triples to ~129 TB/day before compression. Retention length then multiplies this. Derived estimate.
Replication factor
3 replicas per partition
The common production default. With 3 replicas and min.insync.replicas=2 you tolerate one broker loss with no data loss and still accept writes. f+1 replicas tolerate f failures for committed data.
Fan-out
1 write read by N consumer groups
Because consumers track their own offset and reads are non-destructive, one physical copy of a partition serves many groups. A record ingested once can be delivered to analytics, search indexing, and billing independently. Structural, not a measured number.
Retention window
7 days typical, unbounded if compacted
Time based retention of 7 days is a common default for event streams. Compacted topics instead keep the latest value per key indefinitely, so their size tracks key cardinality rather than message volume. Config dependent.
A producer picks a topic and, from a record's key, computes a target partition, usually hash(key) mod partitionCount, so all records for the same key land in the same partition and keep their relative order. The producer batches records per partition, optionally compresses the batch, and sends it to the broker that currently leads that partition. The leader appends the batch to the tail of its local log, which is a set of segment files on disk, and assigns each record the next offset. Follower brokers for that partition continuously fetch from the leader and append the same records to their own logs. Once every replica in the in-sync set has the batch, the record is committed and the high watermark advances; only then does the producer's acks=all request return success and only committed records become visible to consumers. On the read side, consumers in a group are each assigned a subset of the partitions by the group coordinator. A consumer sends fetch requests to the leader of each partition it owns, reads records forward from its last committed offset, and periodically commits its new offset back to Kafka, usually to the internal __consumer_offsets topic. If a consumer dies or a new one joins, the group rebalances and partitions are reassigned. Cluster metadata, which broker leads which partition, the ISR sets, topic configs, is managed by the controller, which in modern Kafka is elected among a small set of controllers that form a KRaft (Raft) quorum rather than living in an external ZooKeeper ensemble.
In a real interview, sketch this on the whiteboard before diving into any single box.
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
A single Kafka server that stores partition data on local disk and serves produce and fetch requests. A cluster is a set of brokers, and each broker leads some partitions and follows others. Brokers are largely stateless about clients, since consumers, not brokers, track read position.
A topic is a named stream, split into partitions for scale. Each partition is an ordered, append-only log addressed by offset, and it is the unit of parallelism, ordering, and replication. Ordering holds within a partition but not across partitions of the same topic.
The client that appends records. It decides the partition from the key, buffers records into per-partition batches, optionally compresses and enables idempotence, and chooses an acks level. Batching is what turns many small writes into a few large sequential disk appends.
A consumer reads records forward from a partition and commits its offset. Consumers join a group so partitions are divided among members, giving scalable, ordered consumption. Different groups reading the same topic are fully independent, which is how one log fans out to many pipelines.
Each partition has one leader and zero or more followers. The leader tracks which followers are caught up, the in-sync replica set, and a write is committed only once all in-sync replicas have it. If the leader fails, a new leader is elected from the ISR so no committed data is lost.
The controller manages cluster metadata: partition leadership, ISR membership, and topic configuration. Historically Kafka stored this in ZooKeeper. Modern Kafka uses KRaft, an internal Raft quorum of controller nodes that keep the metadata log in the __cluster_metadata topic and elect an active controller.
A partition log is split into segment files plus index files that map offsets to physical positions. Writes are sequential appends to the active segment. Retention deletes or compacts old segments, and the sparse index lets a consumer seek to any offset quickly without scanning.
record. The unit stored in a partition log. Offset is assigned by the leader and is unique and monotonically increasing within a partition. Key drives partitioning and…
Every table for Apache Kafka, its keys, and which store fits it and why.
Why a log beats a traditional queue. A classic message queue treats delivery as destructive: a broker holds a message, hands it to one consumer, and deletes it on ack. That makes the broker track…
7 deep dives on the parts that break at scale: Why a log beats a traditional queue; Partitioning for scale and the ordering tradeoff; Replication, ISR, and the acks tradeoff.
Log with consumer-tracked offsets versus broker-tracked per-message acks. The log makes reads non-destructive, so replay and multi-consumer fan-out are trivial and the broker stays simple. The cost is that Kafka does not natively do…
6 decisions an interviewer expects you to defend, each with the alternative and why it loses here.
A consumer charges a customer's card for each record. After a rebalance some records are processed twice. How do you stop double charges?
4 follow-up questions with model answers, the interview scorecard for this question, and a mock interview on Design Apache Kafka with the AI tutor.
Kafka came out of LinkedIn and was described in the 2011 paper "Kafka: a Distributed Messaging System for Log Processing," which already framed it as a commit log built for high-throughput log and event data rather than a traditional broker. LinkedIn has publicly discussed running Kafka at the scale of trillions of messages per day across many clusters. Confluent, founded by Kafka's creators, maintains the canonical design documentation and wrote the definitive engineering posts on exactly-once semantics, the idempotent producer, and transactions. The move off ZooKeeper is real and shipped: KIP-500 introduced the KRaft metadata quorum, and Apache Kafka 4.0 in 2025 removed ZooKeeper support entirely so KRaft is now the only mode. The performance model, sequential I/O, page cache, and zero-copy sendfile, is documented in Kafka's own design pages and is the reason a commodity broker can push hundreds of megabytes per second.
Sources
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Write-Ahead Logging
advanced / consistency models
Leader Election
advanced / distributed systems core
Consensus Algorithms
advanced / distributed systems core
Exactly-Once Semantics
advanced / stream batch processing
Data Partitioning
intermediate / data replication distribution
Fan-Out/Fan-In
intermediate / messaging event systems
Cross-Region Replication
intermediate / data replication distribution
interview readiness
Three questions from this guide. Answer them without scrolling up, the way you would have to in the room.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
NEWSystem Design Hands-On
Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.
Free PDF · 79 pages
25 real tech interview questions, answered in full
Five from each of our AI, GenAI, Inference, Maths for ML and Forward Deployed packs. Each with a diagram and a model answer. Free by email.
free tool
Check these numbers yourself
Put in your own guesses for Apache Kafka and see requests per second, storage, bandwidth and servers, with every sum written out.
Open the Capacity EstimatorIt is really nice and good Please join the course