We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Messaging & Event Systems
A named feed or category to which producers publish records. Topics are split into partitions for parallelism, and each partition is an ordered, immutable log.
In short
A Kafka topic is a named category that producers write records to and consumers read records from. Each topic is split into one or more partitions, and every partition is an ordered, append-only log where records keep the same position forever once written.
In short
A Kafka topic is a named category that producers write records to and consumers read records from. Each topic is split into one or more partitions, and every partition is an ordered, append-only log where records keep the same position forever once written.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Kafka Topic across a run of lessons, not one page. These 4 alone are about 77 minutes of step-by-step reading, every one with a quiz.
Design consumers that produce the same result whether a message is processed once or ten times.
Once you have seen a value, you never see an older one, the minimum useful consistency guarantee.
Scale message processing by having multiple consumers race to handle messages from the same queue.
Detect and isolate messages that crash your consumers, before they crash them forever.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Idempotent Consumerpay once,
yours for life
A topic is the unit of organization in Apache Kafka. You give it a name like orders, payments, or user-clicks, and from then on producers send messages to that name and consumers subscribe to it. Nothing else needs to coordinate. A producer does not know who is reading, and a consumer does not know who is writing. The topic is the shared channel between them.
The important detail is that a topic is not a queue. It is a commit log. When a producer sends a record, Kafka appends it to the end of a log file and assigns it a number called an offset. The record sits there, unchanged, until the topic's retention period expires. Reading a record does not delete it. Ten different consumer groups can read the same orders topic and each one tracks its own position independently.
Because the log is append-only and never edited in place, Kafka can write to it sequentially on disk. Sequential disk writes are close to memory speed, which is a big part of how a single broker handles hundreds of thousands of records per second.
A topic is split into partitions, and each partition is itself an ordered log. Partitions are what give Kafka its scale. If a topic has 12 partitions, up to 12 consumers in one group can read from it in parallel, one per partition. Add partitions and you add throughput.
Ordering is guaranteed inside a single partition but not across the whole topic. Records in partition 0 are read in the exact order they were written, but there is no global order spanning partition 0, 1, and 2. This trade-off is the price of parallelism.
You control which partition a record lands in through its key. Kafka hashes the key and uses the result to pick a partition, so every record with the same key goes to the same partition. If you key by customer ID, all events for one customer stay in order. If you send records with no key, Kafka spreads them across partitions evenly and you give up per-key ordering.
One rule trips people up: you can increase a topic's partition count later, but you cannot decrease it, and adding partitions changes the key-to-partition mapping for new records. Pick a partition count with growth in mind.
Use a separate topic per kind of event, not per consumer. One topic called payments that several services read is the normal pattern. Splitting by department or team usually creates churn. Splitting by event meaning, such as payment-authorized versus payment-refunded, holds up well as the system grows.
Retention is a per-topic setting and a real design choice. The default keeps records for 7 days, after which old log segments are deleted. You can set it to hours for high-volume telemetry, or to forever combined with log compaction when the topic represents the current state of something, like the latest profile per user ID.
The main trade-offs are partition count and replication factor. Too few partitions caps your parallelism. Too many adds open-file and metadata overhead on every broker and slows leader elections. A replication factor of 3 is standard in production so a topic survives losing a broker, but each replica multiplies the disk and network cost.
Picture a ride-hailing app. A topic named driver-locations receives a GPS ping from every active driver every few seconds. You key each record by driver ID, so all pings from one driver land in the same partition and arrive in order. The topic has 50 partitions, so 50 consumers can process location updates at once.
One consumer group powers the live map. A second consumer group, reading the exact same topic from its own offset, feeds a fraud detection model that flags impossible jumps in position. A third group archives everything to cold storage. None of these interfere with each other because each tracks its own offset, and the records themselves are never removed by a read.
If the fraud service crashes for an hour, it simply resumes from its last committed offset when it comes back and replays the missed records. The map service, sitting at a different offset, never noticed. That independence between writers, readers, and the durable log is what makes a topic more than just a message pipe.
Built Kafka and runs thousands of topics carrying activity events, metrics, and logs across trillions of messages a day.
Uber
Uses topics keyed by trip and driver ID to stream location and pricing events into real-time matching and surge pricing.
Netflix
Routes playback and operational events through Kafka topics feeding its real-time monitoring and recommendation pipelines.
Confluent Cloud
Sells managed Kafka where topic creation, partition count, and retention are the core knobs customers configure.
No. A queue typically removes a message once a consumer reads it, and one message goes to one consumer. A Kafka topic keeps every record for its retention period regardless of reads, and any number of consumer groups can read the same records independently. It is a durable log, not a pop-once queue.
Enough to meet your target throughput and your maximum parallel consumers, since one partition is read by at most one consumer per group. Many teams start around 6 to 12 for moderate load. Avoid going wildly high, because thousands of partitions per broker add metadata overhead and slow recovery. You can add partitions later but never remove them.
Only within a single partition. Records in one partition are always read in the order they were written. There is no ordering guarantee across the whole topic. To keep related records ordered, give them the same key so Kafka routes them to the same partition.
Nothing changes. Consuming a record does not delete it. Records stay in the topic until the retention period expires, by default 7 days, or until log compaction removes superseded keys. This is why a consumer can replay history or a new consumer group can read everything from the beginning.
Yes, and that is the normal design. Each service uses its own consumer group with its own offset, so they read the same records without competing. A live dashboard, an analytics job, and an archiver can all consume one topic at their own pace.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Kafka Partition
A subset of a Kafka topic that provides ordering guarantees and parallel processing. Each partition lives on one broker and can be consumed by one consumer per group.
Kafka Consumer Group
A set of consumers that cooperatively read from topic partitions. Each partition is assigned to exactly one consumer in the group, enabling parallel processing.
Pub/Sub
A messaging pattern where publishers send messages to topics, and subscribers receive messages from topics they care about. Publishers don't know who's listening.
Message Queue
A buffer that stores messages between producers and consumers. Messages are processed one by one, in order. Think of it as a to-do list for your services.
Webhook
An HTTP callback triggered by an event. Instead of polling for updates, the source system pushes a notification to your URL when something happens.
This page explains the idea. The course covers it step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
See the System Design Masterclass coursecourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Kafka Topic in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
