Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
What is Kafka?
In short
Apache Kafka is a distributed event streaming platform that stores streams of records in ordered, append-only logs called topics, and lets many producers write and many consumers read those records independently. It is used to build real-time data pipelines and event-driven systems that move millions of messages per second between services.
What Kafka actually is
Kafka is a commit log you can share across an entire company. Producers append records to the end of a log, and consumers read forward through that log at their own pace. The log is durable and persisted to disk, so a record is not deleted the moment someone reads it. By default Kafka keeps data for 7 days, and you can set it to keep data for hours, weeks, or forever.
A topic is a named stream of records, like orders or page-views. Each topic is split into partitions, and each partition is the actual ordered log. Records inside one partition have a strict order and a monotonically increasing number called an offset. Records across different partitions have no global order, which is the trade Kafka makes to scale horizontally.
This is the key mental shift from a normal message queue. In a queue, a message is consumed once and gone. In Kafka, the data sits in the log and many independent consumer groups can read the same records for different purposes. One group feeds a search index, another loads a data warehouse, a third runs fraud checks, all from the same topic without interfering with each other.
How it works under the hood
A Kafka cluster is a set of servers called brokers. Each partition is stored on one broker as the leader and copied to other brokers as followers, controlled by the replication factor. A common production setting is replication factor 3, meaning every record exists on 3 brokers. If a broker dies, a follower is promoted to leader and no data is lost as long as a copy survived.
Producers decide which partition a record goes to. If the record has a key, Kafka hashes the key so all records with the same key land in the same partition and stay in order. That is how you keep all events for one user or one order together. Records with no key are spread across partitions for balance.
Consumers join a consumer group. Kafka assigns partitions to consumers in the group so each partition is read by exactly one consumer in that group. This is how Kafka parallelizes: a topic with 12 partitions can be read by up to 12 consumers at once. Each consumer commits the offset it has processed, so after a restart or crash it resumes from where it left off instead of replaying everything.
Older versions relied on Apache ZooKeeper to track cluster metadata. Modern Kafka replaces ZooKeeper with a built-in protocol called KRaft, so a cluster is now just brokers with no separate coordination service to run.
When to use it and the trade-offs
Reach for Kafka when many systems need the same stream of events, when you want to replay history, or when throughput is high enough that a traditional broker would struggle. It comfortably handles hundreds of thousands to millions of records per second on a modest cluster, and the on-disk log means a slow or offline consumer does not block anyone else.
The cost is operational weight and complexity. You run a cluster, pick partition counts up front (you can add partitions but it disrupts key ordering), tune retention and replication, and monitor consumer lag. For a single service that needs a simple work queue with per-message acknowledgement and routing, RabbitMQ, Amazon SQS, or Redis streams are lighter and easier.
Kafka also gives you at-least-once delivery by default, which means a consumer can see the same record twice after a retry. You either make your processing idempotent or turn on exactly-once semantics with transactions, which adds overhead. And ordering only holds inside a partition, so any design that assumes one global order across a whole topic is wrong.
A concrete example
Imagine a ride-hailing app. Every GPS ping, ride request, and payment becomes a record. Driver location events go to a locations topic partitioned by driver id, so all pings for one driver stay ordered. The matching service reads that topic to pair riders with nearby drivers in real time.
The same locations topic is read by a second consumer group that writes the data into a warehouse for analytics, and a third that powers a live map dashboard. None of them slow each other down because each tracks its own offset in the shared log.
If the analytics job has a bug, the team fixes it and resets the consumer group offset to an earlier point, then reprocesses the last several days straight from Kafka. No upstream system has to resend anything, because the events were never thrown away.
Where it is used in production
Built Kafka internally around 2010 and open-sourced it, now running trillions of messages per day for activity feeds, metrics, and operational data.
Netflix
Uses Kafka as the backbone of its real-time data pipeline, ingesting trillions of events daily for monitoring, recommendations, and stream processing.
Uber
Runs one of the largest Kafka deployments to move trillions of messages per day for trip events, surge pricing, fraud detection, and analytics.
Apache Kafka
The open-source project itself, governed by the Apache Software Foundation, with managed offerings from Confluent and Amazon MSK.
Frequently asked questions
- Is Kafka a message queue or a database?
- Neither exactly. It is a distributed append-only log. It behaves like a message queue in that producers send and consumers receive, but unlike a queue it keeps records after they are read so many consumers can replay the same data. It is not a database because you read it forward by offset, not by arbitrary query.
- What is the difference between Kafka and RabbitMQ?
- RabbitMQ is a traditional broker that pushes messages to consumers and deletes them once acknowledged, with rich routing built in. Kafka stores records in a durable log that consumers pull from at their own pace, and keeps them for replay. Pick RabbitMQ for per-message work queues and complex routing, Kafka for high-throughput event streaming and shared event history.
- What are partitions and why do they matter?
- A partition is the actual ordered log inside a topic. More partitions mean more parallelism because each partition can be read by a separate consumer in a group. Records keep order only within a single partition, and records sharing the same key always go to the same partition, so partitioning is how you balance throughput against ordering guarantees.
- Does Kafka still need ZooKeeper?
- No, not in current versions. Kafka now uses a built-in consensus protocol called KRaft to manage cluster metadata, so you run only brokers. ZooKeeper support has been removed in recent releases, which simplifies operations and improves how fast the cluster recovers.
- What delivery guarantees does Kafka give?
- By default at-least-once, meaning a consumer can occasionally see the same record twice after a retry. You can drop to at-most-once by committing offsets before processing, or achieve exactly-once using idempotent producers and transactions, at some performance cost. Most teams make processing idempotent rather than relying on exactly-once.
Learn Kafka hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Kafka as part of a larger topic.
Apache Avro
Schema-evolution-friendly binary serialization, the serialization format built for big data
intermediate · api design protocols
Schema Registry
A central repository for API schemas, enforce compatibility, enable evolution, prevent breaking changes
intermediate · api design protocols
Change Data Capture (CDC)
Capture every database change as a stream of events, the backbone of modern data pipelines
intermediate · data replication distribution
Pub/Sub Pattern
One publisher, many subscribers, the pattern that powers event-driven systems at scale
intermediate · messaging event systems
Message Ordering
Guaranteeing messages arrive in the right sequence, harder than it sounds
intermediate · messaging event systems
See also
Related glossary terms you might want to look up next.
Pub/Sub
A messaging pattern where publishers send messages to topics, and subscribers receive messages from topics they care about. Publishers don't know who's listening.
Message Queue
A buffer that stores messages between producers and consumers. Messages are processed one by one, in order. Think of it as a to-do list for your services.
Event Sourcing
Storing every state change as an immutable event instead of just the current state. You can rebuild any past state by replaying events.
Webhook
An HTTP callback triggered by an event. Instead of polling for updates, the source system pushes a notification to your URL when something happens.
Kafka Topic
A named feed or category to which producers publish records. Topics are split into partitions for parallelism, and each partition is an ordered, immutable log.
Kafka Partition
A subset of a Kafka topic that provides ordering guarantees and parallel processing. Each partition lives on one broker and can be consumed by one consumer per group.