Apache Kafka System Design Interview: Building a Distributed Message Queue on a Commit Log
A single Kafka cluster at a large company routinely moves trillions of messages a day across thousands of topics. LinkedIn, where Kafka was born, has publicly described clusters handling on the order of 7 trillion messages per day. The trick is that a well tuned broker can sustain hundreds of megabytes per second of writes on commodity disks, because Kafka almost never does a random disk seek. These are published or widely cited figures, and exact throughput depends heavily on hardware, message size, and batching.
Kafka is a distributed, partitioned, replicated commit log dressed up as a messaging system. A topic is split into partitions, and each partition is an append-only, totally ordered sequence of records addressed by a monotonically increasing offset. Producers append to the tail, consumers read forward at their own pace and track their own offset, and the same records can be re-read by many independent consumer groups because reading does not destroy the message. Ordering is guaranteed only within a partition, which is the price you pay for horizontal scale. Durability comes from replicating each partition to several brokers, with one leader taking all reads and writes and followers pulling to stay in sync. The in-sync replica set plus the acks setting lets you dial the tradeoff between latency and how many failures you can survive without losing an acknowledged write. The whole thing is fast because it leans on sequential disk I/O, the OS page cache, and zero-copy transfer rather than clever in-memory structures. Coordination that used to live in ZooKeeper now lives inside Kafka itself through the KRaft metadata quorum.
Asked at: Asked at LinkedIn, Confluent, Uber, Stripe, Datadog, and most companies that run a large event backbone. It also shows up in generic interviews as "design a distributed message queue" or "design a pub-sub system," where Kafka is the reference implementation the interviewer has in mind.
Why this question is asked
This problem tests whether a candidate really understands the log as a primitive, not just an API they call. It forces you to reason about ordering versus parallelism, about the difference between a message being written and a message being committed, and about how replication interacts with availability during a broker failure. A strong candidate will talk about partition keys and hot partitions, about consumer group rebalancing, and about why at-least-once is the honest default and what it actually costs to get closer to exactly-once. It is a good filter because the naive answer, a queue with producers and consumers, falls apart the moment the interviewer asks how you keep order, survive a crash, and still push a million messages a second.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Producers publish records to a named topic, optionally with a key that decides the target partition
- Consumers subscribe to topics and read records in order within each partition, tracking progress by offset
- Support consumer groups so a topic can be load balanced across many consumer instances, each partition owned by exactly one member at a time
- Retain messages for a configurable time or size window so late or replaying consumers can re-read history
- Support multiple independent consumer groups reading the same topic without interfering with each other
- Let producers choose a durability level per request, from fire and forget to wait for the full in-sync replica set
- Support log compaction so a topic can keep only the latest value per key as a changelog
- Allow topics to be created, partitioned, and have their replication factor set, and allow partitions to be added later
Non-functional requirements
- High write throughput, on the order of hundreds of thousands to millions of messages per second per cluster (depends on hardware and batching)
- Low end to end latency, single digit to low double digit milliseconds for a well tuned pipeline
- Durability: an acknowledged write with acks=all must survive the failure of up to f brokers given f+1 in-sync replicas
- High availability: a partition should keep serving through a single broker failure via automatic leader failover
- Horizontal scalability by adding brokers and partitions without downtime
- Ordering guarantee within a partition, and no silent message loss for committed records
- Predictable performance under retention, so old data ages out without slowing the write path
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Peak ingest
~1M messages/sec, ~500 MB/sec per cluster
Assume 1M msgs/sec at an average 500 bytes each. 1M * 500 bytes = 500 MB/sec of raw writes. Batching and compression cut the bytes on the wire and to disk well below this. Derived example, not a fixed limit.
Partitions for parallelism
~1,000 partitions for a busy topic
Consumer parallelism is capped by partition count, since one partition maps to one consumer per group. To let 1,000 consumers work in parallel you need at least 1,000 partitions. Derived from the one-partition-per-consumer rule.
Storage per day
~43 TB/day raw before replication and compression
500 MB/sec * 86,400 sec = ~43 TB/day of raw ingest. With replication factor 3 the on-disk cost triples to ~129 TB/day before compression. Retention length then multiplies this. Derived estimate.
Replication factor
3 replicas per partition
The common production default. With 3 replicas and min.insync.replicas=2 you tolerate one broker loss with no data loss and still accept writes. f+1 replicas tolerate f failures for committed data.
Fan-out
1 write read by N consumer groups
Because consumers track their own offset and reads are non-destructive, one physical copy of a partition serves many groups. A record ingested once can be delivered to analytics, search indexing, and billing independently. Structural, not a measured number.
Retention window
7 days typical, unbounded if compacted
Time based retention of 7 days is a common default for event streams. Compacted topics instead keep the latest value per key indefinitely, so their size tracks key cardinality rather than message volume. Config dependent.
High-level architecture
A producer picks a topic and, from a record's key, computes a target partition, usually hash(key) mod partitionCount, so all records for the same key land in the same partition and keep their relative order. The producer batches records per partition, optionally compresses the batch, and sends it to the broker that currently leads that partition. The leader appends the batch to the tail of its local log, which is a set of segment files on disk, and assigns each record the next offset. Follower brokers for that partition continuously fetch from the leader and append the same records to their own logs. Once every replica in the in-sync set has the batch, the record is committed and the high watermark advances; only then does the producer's acks=all request return success and only committed records become visible to consumers. On the read side, consumers in a group are each assigned a subset of the partitions by the group coordinator. A consumer sends fetch requests to the leader of each partition it owns, reads records forward from its last committed offset, and periodically commits its new offset back to Kafka, usually to the internal __consumer_offsets topic. If a consumer dies or a new one joins, the group rebalances and partitions are reassigned. Cluster metadata, which broker leads which partition, the ISR sets, topic configs, is managed by the controller, which in modern Kafka is elected among a small set of controllers that form a KRaft (Raft) quorum rather than living in an external ZooKeeper ensemble.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Broker
A single Kafka server that stores partition data on local disk and serves produce and fetch requests. A cluster is a set of brokers, and each broker leads some partitions and follows others. Brokers are largely stateless about clients, since consumers, not brokers, track read position.
Topic and partition
A topic is a named stream, split into partitions for scale. Each partition is an ordered, append-only log addressed by offset, and it is the unit of parallelism, ordering, and replication. Ordering holds within a partition but not across partitions of the same topic.
Producer
The client that appends records. It decides the partition from the key, buffers records into per-partition batches, optionally compresses and enables idempotence, and chooses an acks level. Batching is what turns many small writes into a few large sequential disk appends.
Consumer and consumer group
A consumer reads records forward from a partition and commits its offset. Consumers join a group so partitions are divided among members, giving scalable, ordered consumption. Different groups reading the same topic are fully independent, which is how one log fans out to many pipelines.
Replication and the ISR set
Each partition has one leader and zero or more followers. The leader tracks which followers are caught up, the in-sync replica set, and a write is committed only once all in-sync replicas have it. If the leader fails, a new leader is elected from the ISR so no committed data is lost.
Controller and KRaft quorum
The controller manages cluster metadata: partition leadership, ISR membership, and topic configuration. Historically Kafka stored this in ZooKeeper. Modern Kafka uses KRaft, an internal Raft quorum of controller nodes that keep the metadata log in the __cluster_metadata topic and elect an active controller.
Storage engine (log segments)
A partition log is split into segment files plus index files that map offsets to physical positions. Writes are sequential appends to the active segment. Retention deletes or compacts old segments, and the sparse index lets a consumer seek to any offset quickly without scanning.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
recordoffsetkeyvaluetimestampheadersThe unit stored in a partition log. Offset is assigned by the leader and is unique and monotonically increasing within a partition. Key drives partitioning and compaction. Records are immutable once written.
partition_metadatatopicpartition_idleader_brokerreplica_setisrhigh_watermarkHeld by the controller and cached on brokers. ISR is the subset of replicas currently caught up to the leader. High watermark is the highest offset that is committed and therefore readable by consumers.
log_segmentbase_offsetlog_fileoffset_indextime_indexsegment_sizePhysical file layout for a partition. base_offset names the segment; the offset index maps a logical offset to a byte position for fast seeks. Old segments are deleted or compacted by retention.
consumer_offsetsgroup_idtopicpartitioncommitted_offsetcommit_timestampStored in the internal compacted topic __consumer_offsets. Records where each group has read up to per partition, so consumption resumes correctly after a restart or rebalance.
producer_stateproducer_idproducer_epochpartitionlast_sequenceKept per partition to support the idempotent producer. The broker deduplicates by producer id, epoch, and per-partition sequence number, so a retried batch is written at most once.
cluster_metadatametadata_offsetrecord_typebroker_registrationstopic_configsleader_assignmentsThe KRaft metadata log in the __cluster_metadata topic. Controllers replicate it via Raft; the active controller appends changes, followers replicate for redundancy and fast failover.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Why a log beats a traditional queue
A classic message queue treats delivery as destructive: a broker holds a message, hands it to one consumer, and deletes it on ack. That makes the broker track per-message, per-consumer state and makes replay awkward. Kafka flips the model. A partition is an append-only log, and a consumer is just a cursor holding an offset into that log. Reading does not remove anything, so ten different consumer groups can read the same partition at ten different positions with zero coordination. Retention, not consumption, decides when data leaves. This is what lets Kafka serve real-time consumers and batch jobs and a brand new consumer that wants to replay last week, all from one copy of the data. It also simplifies the broker enormously, because the expensive per-consumer bookkeeping moves to a single committed offset per group.
Partitioning for scale and the ordering tradeoff
Throughput and parallelism come from splitting a topic into partitions that live on different brokers. But Kafka only guarantees order within a partition, never across partitions, so partitioning is a deliberate tradeoff between scale and ordering. The lever is the partition key. Records with the same key go to the same partition, so if you key by user id, all events for one user stay ordered while different users spread across the cluster. Choose the key badly and you get hot partitions, where one key, say a single huge customer, funnels a disproportionate share of traffic to one broker while others sit idle. Partition count also bounds consumer parallelism, since a partition is owned by exactly one consumer in a group, so you size partitions for your peak consumer fan-out. You can add partitions later, but that changes the hash mapping for keyed data and breaks the per-key ordering guarantee for existing keys, so it is not a decision to take lightly.
Replication, ISR, and the acks tradeoff
Each partition is replicated to several brokers, one leader and some followers. All produce and fetch traffic goes through the leader; followers pull from the leader and append the same records. The leader tracks the in-sync replica set, the followers that are caught up within replica.lag.time.max.ms, and a record is committed only once every in-sync replica has it. The producer picks how much durability it wants with acks. acks=0 is fire and forget, fastest and least safe. acks=1 waits for the leader's local write only, so a leader crash right after the ack can lose the record. acks=all waits for the full ISR, so with three replicas and min.insync.replicas set to two, the write survives a single broker loss. The docs put it simply: with f+1 replicas a topic tolerates f failures without losing committed messages. This is a primary-backup scheme, not a majority-vote quorum, which is why Kafka can commit with all in-sync replicas rather than a bare majority.
Leader failover and unclean leader election
When a leader broker dies, the controller picks a new leader for each affected partition from the in-sync replica set, because any ISR member is guaranteed to hold every committed record. Producers and consumers transparently discover the new leader and continue. The hard case is when every in-sync replica is unavailable at once. Now you choose between two bad options. Wait for an original ISR member to come back, which preserves consistency but stalls the partition, possibly for a long time. Or enable unclean leader election, which promotes a lagging out-of-sync replica so the partition keeps serving, at the cost of dropping the records that replica never received. Kafka defaults to the safe choice, unclean.leader.election.enable false, so it favors not losing committed data over staying available when the ISR is empty. This is the CAP tradeoff made concrete at the partition level.
Delivery semantics: at-least-once, at-most-once, exactly-once
The honest default is at-least-once. A producer that does not get an ack retries, which can write a duplicate; a consumer that processes a batch and crashes before committing its offset reprocesses on restart. At-most-once is the mirror image: commit the offset first, and a crash means you skip records but never duplicate them. Exactly-once is the hard one, and Kafka builds it in two parts. First, the idempotent producer: the broker assigns each producer an id and a per-partition sequence number, so a retried batch is deduplicated and written once even across retries. Second, transactions: a producer can atomically write to multiple partitions and commit its consumer offsets in the same transaction, so a read-process-write loop either fully commits or is fully rolled back, and consumers set to read_committed never see aborted data. Together these give exactly-once for stream processing within Kafka, which is what the Kafka Streams API relies on. It is worth being precise in an interview: exactly-once holds for Kafka-to-Kafka processing, not automatically for arbitrary external side effects.
The storage layer: sequential writes, page cache, and zero-copy
Kafka's speed is mostly a story about not fighting the operating system. Writes are sequential appends to the active log segment, so even spinning disks give high throughput because there are no random seeks. Kafka does not maintain its own in-memory cache of messages; it writes to the OS page cache and lets the kernel handle flushing and caching, which means a broker with lots of RAM serves recent data straight from memory without Kafka duplicating it. On the read path Kafka uses zero-copy, the sendfile system call, to move bytes from the page cache to the network socket without copying them through user space or re-serializing them, which is a big win when many consumers read the same recent data. Segments plus sparse offset indexes let a consumer seek to any offset cheaply. Retention works at the segment level: whole old segments are deleted or compacted, so aging out data never touches the hot write path. Compaction is the alternative to time-based deletion, keeping only the latest value per key so a topic can act as a durable changelog.
Coordination: from ZooKeeper to KRaft
Kafka needs a source of truth for metadata: which broker leads each partition, who is in each ISR, what topics and configs exist. For most of its life Kafka stored this in an external ZooKeeper ensemble, and a single elected controller broker watched ZooKeeper and pushed changes to the rest of the cluster. That worked but added an operational dependency and made metadata a scaling bottleneck, because propagating changes for very large clusters was slow. KIP-500 replaced ZooKeeper with KRaft, a self-managed metadata quorum. A small set of controller nodes run an event-driven variant of the Raft consensus protocol, storing all metadata as an ordered log in the internal __cluster_metadata topic. One controller is the active leader and appends changes; the others replicate the log. Brokers become simple followers of that metadata log, which makes failover faster and lets clusters hold far more partitions. As of Kafka 4.0, released in 2025, ZooKeeper mode is removed entirely and KRaft is the only supported mode.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Log with consumer-tracked offsets versus broker-tracked per-message acks
The log makes reads non-destructive, so replay and multi-consumer fan-out are trivial and the broker stays simple. The cost is that Kafka does not natively do per-message acknowledgment or selective redelivery the way RabbitMQ does, so patterns like a dead-letter on a single bad message take extra work.
Partition-level ordering versus global ordering
Ordering only within a partition is what allows horizontal scale across brokers. If you truly need total order for a topic you are forced to a single partition, which caps throughput at one broker. Most designs key by entity id to get the ordering that actually matters and accept no global order.
acks=all durability versus acks=1 latency
acks=all waits for the whole in-sync set, so it survives broker loss but adds the slowest replica's latency to every write. acks=1 acknowledges on the leader alone for lower latency but can lose data if the leader dies before followers catch up. This is the core durability versus latency dial.
Primary-backup ISR replication versus majority-vote quorum
Kafka commits when all in-sync replicas have the write, not just a majority. This tolerates f failures with f+1 replicas instead of the 2f+1 a quorum system needs, so it is more storage efficient, but it depends on the ISR shrinking correctly and on min.insync.replicas to avoid committing to a lone leader.
Unclean leader election off versus on
Leaving it off preserves committed data but can stall a partition when the entire ISR is down. Turning it on keeps the partition available by promoting a stale replica, at the risk of silently dropping records. It is a consistency versus availability choice you make per topic.
Exactly-once via idempotence and transactions versus simple at-least-once
Exactly-once removes duplicates and gives atomic multi-partition writes, but it adds producer coordination, transaction markers, and read_committed overhead, and only covers Kafka-to-Kafka flows. Many pipelines are better served by at-least-once plus idempotent consumers, keying downstream writes so duplicates are harmless.
How Apache Kafka actually does it
Kafka came out of LinkedIn and was described in the 2011 paper "Kafka: a Distributed Messaging System for Log Processing," which already framed it as a commit log built for high-throughput log and event data rather than a traditional broker. LinkedIn has publicly discussed running Kafka at the scale of trillions of messages per day across many clusters. Confluent, founded by Kafka's creators, maintains the canonical design documentation and wrote the definitive engineering posts on exactly-once semantics, the idempotent producer, and transactions. The move off ZooKeeper is real and shipped: KIP-500 introduced the KRaft metadata quorum, and Apache Kafka 4.0 in 2025 removed ZooKeeper support entirely so KRaft is now the only mode. The performance model, sequential I/O, page cache, and zero-copy sendfile, is documented in Kafka's own design pages and is the reason a commodity broker can push hundreds of megabytes per second.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Apache Kafka.