Kafka Partitions and Consumer Groups: Who Reads What, and Why
Follow an online shop's order stream through a real Kafka 4.1 broker: where each keyed order lands, who reads it, and what happens when you add more readers than partitions.
- Step 1 of 14
One stream, many readers
An online shop records every order event: placed, paid, shipped. It sends them to Kafka, a system that stores streams of events so other programs can read them. Several teams need those events. Billing must charge each order once, never twice. Analytics wants to count all of them. And at the busy hour there are more orders than one program can handle. Two Kafka ideas cover all of that: partitions and consumer groups.
The drawing
A shop writes order events to one stream. Billing and analytics both need them, and one reader cannot keep up. or press Enter - Step 2 of 14
A topic is split into partitions
Kafka keeps events in a named stream called a topic, here orders. A topic is split into partitions, and each partition is an append-only log: a new event always goes at the end and gets the next number, its offset.
Our topic has 6 partitions. Kafka runs on several servers called brokers, and different partitions can live on different brokers, which is how one topic outgrows one machine.
- Step 3 of 14
The key picks the partition
The program that writes events is the producer. An event can carry a key, such as the customer id. The producer hashes the key and uses the result to pick a partition, so every event with the same key lands in the same partition. Events with no key are spread across partitions.
We sent 12 orders from 6 customers to a real broker. All three of cust-7's orders went to partition 1, at offsets 0, 2 and 5.
Quick checkWhy would the shop key its order events by customer id?
- Step 4 of 14
Order is per partition, not per topic
A program that reads events is a consumer. Kafka promises that a consumer reads one partition's events in exactly the order they were written.
Across partitions there is no order at all. An event written to partition 3 a second after one in partition 0 can still be read first. Order is a promise about one partition, never about the whole topic.
Quick checkOrder 1001 is 'placed', then 'paid'. The producer sends both events with no key. Could billing see 'paid' first?
- Step 5 of 14
A consumer group shares the work
Billing runs three copies of its program, called workers. All three start with the same group id, billing. Kafka treats them as one consumer group, and each worker is a member.
Kafka divides the partitions among the members, and each partition goes to exactly one member. So each event reaches one billing worker, not three. With 3 workers and 6 partitions, each usually gets 2.
- Step 6 of 14
Each group remembers its place
How does a worker know where it stopped? After handling some events, it reports its place back to Kafka. This is a commit, and the saved number is the committed offset: the next event to read.
After the 12 orders, our billing group had committed offset 2 on partition 0 and 10 on partition 1. A restarted worker carries on from there. Events it handled but never committed are read again.
What happens next?The billing worker that owns partitions 4 and 5 crashes. What happens to those partitions?
- Step 7 of 14
More workers than partitions
The busy hour arrives. To keep up, the billing team scales from 3 workers to 8, all in the same group.
The topic still has 6 partitions, and the rule from two steps back still holds: inside one group, a partition belongs to exactly one member.
Predict, then seeSix partitions, eight consumers in one group. How many consumers sit idle?
- Step 8 of 14
Measured on a real broker
We started an Apache Kafka 4.1 broker, created the 6-partition topic and launched 8 consumers with the group id billing. Kafka's group report listed six members with one partition each and two with none.
The idle two are not useless: if a busy one dies, they can take over its partition. But they add no speed. The partition count caps how many members of a group can work at once.
- Step 9 of 14
Adding partitions moves keys
So choose the partition count for the most workers you will ever want in one group. You can add partitions later, but Kafka cannot remove them, and adding them changes where keys go. Events already written stay in their old partition.
We grew the topic from 6 partitions to 8 and sent the same customers again. Five of the six customer keys now landed in a different partition.
Quick checkWhy can adding partitions break per-customer order for a while?
- Step 10 of 14
A second group reads everything again
Now analytics subscribes with its own group id. It does not compete with billing: each group receives every event and keeps its own offsets. A brand-new group starts at the newest events unless told to start from the earliest.
We started analytics from the earliest offset, and it read all 12 orders after billing had already consumed them. Add a fraud team tomorrow and it gets its own group, with no change to billing.
Quick checkBilling has 6 workers and analytics has 1. In normal running, how many times is each event processed in total?
- Step 11 of 14
Rebalancing has a cost
Every time a member joins or leaves, the group shares out the partitions again. This is a rebalance. With the classic protocol's default settings, every member first gives up all its partitions, so the whole group pauses until the new assignment starts.
Newer options shrink that pause: cooperative rebalancing moves only the partitions that change hands, and Kafka 4.0 made a new rebalance protocol, run by the brokers, generally available.
- Step 13 of 14
Sizing it in an interview
This turns into a quick sizing habit. Measure how many events one worker handles per second, divide the peak by it, and that is the fewest workers you need.
Each worker needs its own partition, so that is also the fewest partitions. Then add room to grow, because partitions cannot be removed later.
Predict, then seeOne billing worker handles 500 orders a second. Peak is 4,000 a second. The topic has 6 partitions, spread evenly. What is the most the billing group can handle?
- Step 14 of 14
What every Kafka design starts from
The key picks the partition, and a partition is read in the order it was written, so one customer's events stay in sequence.
Inside a group, each partition has one owner and one committed offset. Members beyond the partition count get nothing to do. A second group gets its own copy of every event and its own offsets, without slowing the first.
What to remember
- Same key, same partition: cust-7's orders all landed in partition 1, in order.
- In a consumer group each partition has one owner, so 8 consumers on 6 partitions left 2 idle.
- Each group keeps its own offsets and reads every event, and partitions cannot be removed later.
Design the consumer side of an order stream at 4,000 events a second where each customer's events must be processed in order. How many partitions, and what happens when a consumer dies?
Order per customer means keying every event by customer id. Same key, same partition, and Kafka orders events within a partition, so one customer's events are processed in sequence by one consumer at a time.
For the count, I measure one consumer's throughput. If it handles 500 events a second, 4,000 needs at least 8 consumers in the group, and each needs its own partition, so at least 8 partitions. I would choose more, say 12 or 16, because partitions cannot be removed and adding them later moves keys to new partitions.
When a consumer dies, the group rebalances and its partitions go to the surviving members, who resume from the committed offsets. Anything processed but not yet committed is processed again, so the consumer must be idempotent, for example by recording the order id it has already charged.
I would also watch for a hot key: one huge customer still lands in one partition, so its lag, how far its consumer has fallen behind, shows up there first.
Follow-up: What if I start more consumers than partitions?
The extra consumers get no partitions and sit idle. On our broker, 8 consumers on 6 partitions gave six members one partition each and two members none.
Follow-up: Can two services both read every event?
Yes. Give each service its own group id. Each group reads every event and keeps its own offsets.
Follow-up: Why not one partition, for total order?
Then only one consumer in the group can work, so throughput is capped at one consumer's speed. Most systems need order per entity, not across everything.
Learn it live
System Design Interview Bootcamp
Its messaging weekend covers partitions, consumer groups and offsets, follows one message through Kafka while a broker fails, then designs a Kafka-like queue on the whiteboard.
See the System Design Interview BootcampWhat this concept covers
- One stream, many readers
- A topic is split into partitions
- The key picks the partition
- Order is per partition, not per topic
- A consumer group shares the work
- Each group remembers its place
- More workers than partitions
- Measured on a real broker
- Adding partitions moves keys
- A second group reads everything again
- Rebalancing has a cost
- The exception: share groups
- Sizing it in an interview
- What every Kafka design starts from
Questions people ask
- What is a Kafka partition?
- An ordered, append-only log that holds part of a topic's events. Each event in it has an offset, and a topic's partitions can live on different brokers.
- What happens if a consumer group has more consumers than partitions?
- The extra consumers get no partition and sit idle. On a real Kafka 4.1 broker, 8 consumers in one group on 6 partitions left 2 consumers with nothing to read.
- Does Kafka guarantee message order?
- Only within a partition. Events with the same key go to the same partition, so they stay in order. There is no order across partitions.
- Can two consumer groups read the same topic?
- Yes. Each group receives every event and keeps its own committed offsets, so billing and analytics can both read the full stream. A new group starts from the newest events unless it is set to start from the earliest.
Where the facts come from
Every number on this page was measured on a real run (scripts/labs/concepts/kafka-groups.sh in our repository) or is quoted from the official documentation below.