Kafka

Kafka is a commit log you can share across an entire company. Producers append records to the end of a log, and consumers read forward through that log at their own pace. The log is durable and persisted to disk, so a record is not deleted the moment someone reads it. By default Kafka keeps data for 7 days, and you can set it to keep data for hours, weeks, or forever.

A topic is a named stream of records, like orders or page-views. Each topic is split into partitions, and each partition is the actual ordered log. Records inside one partition have a strict order and a monotonically increasing number called an offset. Records across different partitions have no global order, which is the trade Kafka makes to scale horizontally.

This is the key mental shift from a normal message queue. In a queue, a message is consumed once and gone. In Kafka, the data sits in the log and many independent consumer groups can read the same records for different purposes. One group feeds a search index, another loads a data warehouse, a third runs fraud checks, all from the same topic without interfering with each other.

Reach for Kafka when many systems need the same stream of events, when you want to replay history, or when throughput is high enough that a traditional broker would struggle. It comfortably handles hundreds of thousands to millions of records per second on a modest cluster, and the on-disk log means a slow or offline consumer does not block anyone else.

The cost is operational weight and complexity. You run a cluster, pick partition counts up front (you can add partitions but it disrupts key ordering), tune retention and replication, and monitor consumer lag. For a single service that needs a simple work queue with per-message acknowledgement and routing, RabbitMQ, Amazon SQS, or Redis streams are lighter and easier.

Kafka also gives you at-least-once delivery by default, which means a consumer can see the same record twice after a retry. You either make your processing idempotent or turn on exactly-once semantics with transactions, which adds overhead. And ordering only holds inside a partition, so any design that assumes one global order across a whole topic is wrong.