Kafka Partition

A Kafka topic is a logical name like orders or page-views. The real storage lives in partitions. When you create a topic you pick a partition count, for example 12, and Kafka spreads those 12 partitions across the brokers in the cluster. Each partition is an independent log file on disk that only ever grows at the end.

Every message in a partition gets a monotonically increasing number called the offset, starting at 0. Offset 0, then 1, then 2, and so on. The offset is unique only within that one partition, not across the whole topic. A consumer tracks its position by remembering the offset it has read up to.

Ordering is the key property. Kafka guarantees order inside a single partition and nowhere else. If you publish messages A, B, C to the same partition, every consumer reads them as A, B, C. If A, B, C land on three different partitions, you get no ordering guarantee between them.

When a producer sends a message, it decides which partition to write to. If the message has a key, Kafka hashes the key and takes the result modulo the partition count, so all messages with the same key always go to the same partition. If there is no key, the producer spreads messages across partitions to balance load. This is why you key by something like user_id or order_id when you need per-entity ordering.

Each partition has one leader broker that handles all reads and writes, plus follower replicas on other brokers that copy the data. The replication factor, often 3, decides how many copies exist. If the leader broker dies, one of the in-sync followers is promoted to leader, so the partition stays available.

On the read side, consumers in the same group split the partitions between them. With 12 partitions and 4 consumers, each consumer owns 3 partitions. A single partition is never read by two consumers in the same group at once, which is what preserves order. That also means your maximum parallelism for one group equals the partition count: 12 partitions cap you at 12 active consumers.

More partitions means more parallelism and higher throughput, because work spreads across more brokers and more consumers. If one topic needs to handle 500,000 messages per second, you give it enough partitions that no single broker is overwhelmed.

But partitions are not free. Each one is open file handles, memory, and replication traffic on the brokers. A cluster with hundreds of thousands of partitions takes longer to recover after a broker failure and adds latency to leader elections. Confluent has long suggested keeping a rough ceiling in the low tens of thousands of partitions per broker as a planning guideline.

The hard rule that bites people: you can increase a topic's partition count later, but you cannot decrease it, and increasing it breaks key-based ordering for existing keys because the hash-to-partition mapping changes. So pick a count that fits your expected peak throughput from the start, and over-provision a little rather than guess too low.

Picture a ride-hailing app with a topic called trip-events and 24 partitions. Each event is keyed by trip_id. Every event for trip 8842, requested, driver-assigned, started, completed, always lands on the same partition, so a consumer processing that trip sees the lifecycle in the correct order.

The consumer group billing-service has 24 instances, one per partition, so all 24 process trips in parallel. A separate group analytics reads the same 24 partitions independently with its own offsets, because each group tracks its own position. The two groups do not interfere.

If billing-service falls behind, you can scale it only up to 24 instances. Adding a 25th instance leaves it idle because there is no 25th partition to give it. That is the moment teams discover their partition count was the real throughput limit, not the number of servers.