Kafka vs RabbitMQ

Three shapes, not two products

First, leave the product names aside. There are three common ways to move messages between programs, and every broker is built around one or two of them.

A message queue is a to-do list. A producer adds a job, and exactly one worker takes it and does it. Add more workers and they share the jobs. Resize this photo, send this email, make this PDF.

Publish and subscribe (pub/sub) is a broadcast. A producer sends one event, and every subscriber (every program that asked for these events) gets its own copy. A customer buys something: the email service sends a receipt, the shop's item count goes down, and the reporting team counts the sale.

An event log is a notebook you only add to. Producers write at the end. The log keeps messages for a set time, and each reader keeps its own place. A reader can go back and read again.

RabbitMQ is built around queues. It can also send a copy to many readers (pub/sub). Kafka is built around the log, and it can also act like the other two. Today each one can do a little of the other's job: RabbitMQ has streams, which are logs, and Kafka 4.2 made share groups ready for real use, and they work like a queue. But the main design of each product is the same as before, and that main design is what the rest of this page is about.

How RabbitMQ routes a message

In RabbitMQ a producer never writes straight into a queue. It sends the message to an exchange, with a short label called a routing key, for example 'order.eu.created'. Bindings are rules that connect an exchange to queues. The exchange reads the key, checks the bindings, and puts a copy in every queue that matches.

There are four main kinds of exchange (RabbitMQ also has a few special ones, such as local random and modulus hash). A direct exchange sends to queues whose binding key is exactly the same as the routing key. A topic exchange matches patterns: * stands for exactly one word and # for zero or more words, so 'order.eu.*' catches 'order.eu.created' but not 'order.us.created'. A fanout exchange ignores the key and copies to every bound queue. A headers exchange looks at message headers instead of the key.

This is why RabbitMQ feels flexible. The producer does not know who reads. Someone can add a queue, bind it with a pattern, and start getting part of the messages without changing the producer's code.

RabbitMQ also gives you control over each single message. A worker acks each message when its job is done; until then, RabbitMQ keeps it and can give it to someone else. Prefetch, set with a command called basic.qos, limits how many unfinished messages one worker may hold, so a worker is never flooded. A message can have a time to live (TTL), after which it expires. A message that is rejected, expires or fails too many times can go to a dead-letter exchange, which puts it aside in a separate queue. RabbitMQ 4.3 also gave quorum queues strict priorities: 32 levels, and important messages always go first, so a steady flow of urgent work can stop the rest from ever being handled. It also added an optional delayed retry, where each new try waits a little longer.

One trap: if a message matches no binding, RabbitMQ drops it by default. Set the mandatory flag so the publisher is told, or give the exchange an alternate exchange that catches unmatched messages. Publisher confirms will not warn you, because they only say the broker received the message.

All of this is record keeping for every single message. It is what makes RabbitMQ good at jobs. It is also extra work, and that extra work limits how fast one queue can go.

How Kafka stores a message

A Kafka topic is split into partitions. Each partition is a log: messages are only ever added at the end, and each one gets the next number, its offset.

Which partition does a message go to? If it has a key, the default partitioner hashes the key, so the same key always lands in the same partition. With no key, the producer fills one partition for a while and then moves on (the docs call it the sticky partition).

Reading does not delete anything. A consumer group, a team of readers working together, remembers how far it has read by saving its offset in an internal topic called __consumer_offsets. A second group keeps its own offset on the same data.

Kafka keeps the data, and each group keeps its own bookmark. Because of this, Kafka can do three things a queue cannot do. Many groups read the same messages, and none of them slows down another. A group can rewind and read again. A brand-new reader can start from the oldest message still kept and build its own copy of the history.

Consumer groups, and the limit of one reader per partition

Inside one consumer group, each partition is read by exactly one member at a time. So the most readers a group can use is the number of partitions. A topic with 3 partitions can keep 3 readers busy. A fourth reader gets nothing. Our test below shows exactly this.

When a reader joins, leaves or stops responding, Kafka moves partitions between the members. This is called a rebalance. With the default assignor (range), which is the rule that decides which reader gets which partition, the whole group pauses and no member reads until the move is done. That pause is why teams watch rebalances closely. The cooperative assignor is also in the default list and moves fewer partitions, but it is only used once you remove range from the list. Since Kafka 4.0 there is a new way to rebalance, ready for general use (written up as KIP-848, a Kafka Improvement Proposal). It moves only the partitions that must move. The server supports it by default, but each reader must turn it on with the setting group.protocol=consumer; if it does not, Kafka 4.3 still uses the old way. The docs plan to make the new way the default in Kafka 5.0.

Kafka also has a second way to read: share groups, also called Queues for Kafka. They first appeared in Kafka 4.0 and became ready for real use in 4.2. In a share group, more readers than partitions can work at once, every message is acked on its own, and Kafka counts how many times a message was handed out (5 tries by default). It works on normal topics; Kafka did not add a separate queue object. Each message is locked to one reader for 30 seconds by default, so a slower job must renew its lock or the message goes to another reader. But you lose order: Kafka keeps the order only inside one batch of messages that a reader gets.

Order: inside a partition, never across

Kafka does not keep one order for a whole topic. It keeps order inside each partition. Because the same key always goes to the same partition, all messages for one customer or one order id are read in the order they were written. Pick the key to be the thing whose order matters.

What you do not get is order between keys in different partitions. In our test, the first messages a reader got were numbers 2, 8, 9, 12, 18, 19: all from the customers that hash to one partition, long before message 0 arrived from another partition. Every customer's own messages were still in order.

If you truly need one order for everything, you need one partition, and so one reader in a group. This is the cost: one order for everything means less work done at the same time.

Sometimes a producer sends a batch of messages again, because it did not hear back the first time. A setting called the idempotent producer (explained in the next part) makes sure that batch is not saved twice or in the wrong place. It is on by default (enable.idempotence=true) in current Kafka. One more thing: the partition comes from the key and the number of partitions, so adding partitions later sends many keys to a different partition. For a while, a key's old messages sit in one partition and its new ones in another, and a reader can see the new ones first. Choose the number of partitions at the start.

RabbitMQ queues are first in, first out, but with several workers, retries and requeues, messages can finish in a different order. If order matters per key there, use one worker per group of keys, or the single active consumer setting, which lets only one worker read a queue at a time.

Delivery: losing a message or doing it twice

'Make it reliable' is a choice, not a single setting. The choice is about what happens if a reader crashes in the middle of a job. It depends on one thing: does the reader save its progress before or after it does the work?

At most once: save the bookmark first, then work. If it crashes after saving and before finishing, that message is lost. Nothing is ever done twice. This is fine for metrics (numbers that measure the system, like page views) where a small gap does not matter.

At least once: work first, then save. If it crashes after the work and before saving, the message comes again and is done twice. Nothing is lost. This is the usual choice, and it is safe when the work is idempotent. RabbitMQ works the same way: the ack is the 'save' step, so a worker acks after the job, and a message that was never acked is delivered again. One limit for long Kafka jobs: a reader that does not call poll() within max.poll.interval.ms (5 minutes by default) is treated as dead, and its partitions move to another reader.

Watch the Kafka default here. enable.auto.commit is true, so the consumer saves its offset on a timer (every 5 seconds) while you call poll(). If your code handles each message before the next poll(), that is at least once. If it hands messages to other threads and polls again, an offset can be saved before the work is done, and that is at most once.

Exactly once in Kafka needs two parts. The idempotent producer gives each producer an id and a sequence number, so the broker drops a duplicate send. A transaction is a group of steps where either all of them happen or none of them do. In Kafka, a program reads some messages, then one transaction writes its results to other topics and saves the bookmark of what it read. Readers must set isolation.level=read_committed (the default is read_uncommitted) to skip aborted writes.

The limit that is easy to miss: this is exactly once from Kafka to Kafka. A row written to Postgres or an email sent is outside the transaction, and Kafka cannot undo it. For those, the writer on the other side must be idempotent.

Keeping copies: Kafka's ISR and RabbitMQ's quorum queues

Both systems copy data to other servers so one broken server does not lose messages. They do it in two different ways, and the difference is a good one to understand for any distributed system.

In Kafka, each partition has one leader (the copy that takes writes) and some followers that copy from it. The in-sync replica set (ISR) is the list of copies that have every message the leader has. With acks=all (a producer setting, and the default), a write counts only when every member of the ISR has it. The problem is that the ISR can shrink. If followers fall behind, it can shrink to the leader alone. So Kafka has a lower limit, a setting called min.insync.replicas. Its default is 1, which allows that. The docs' usual safe setup is 3 copies, min.insync.replicas=2 and acks=all. Then, if fewer than 2 copies are in sync, Kafka refuses the write instead of risking it.

Kafka also does not force each write onto the disk (fsync) before it answers; its design doc says doing that on every write would cost too much speed. It relies on the copies instead. So put the copies on different racks or availability zones (separate parts of a data center, with their own power), so one power cut cannot hit them all.

unclean.leader.election.enable decides what happens when no in-sync copy is left. It is false by default: Kafka waits rather than letting an out-of-date copy become leader and throw away writes it never got. That is choosing correct data over being available, in one setting.

RabbitMQ's quorum queues use Raft. A queue has members, 3 by default, one leader and the rest followers. A message is confirmed to the publisher only after a majority of members have written it: 2 of 3, or 3 of 5. Odd numbers are recommended: with 4 members, a split into 2 and 2 leaves no side with a majority.

Why is a majority safe? Take 3 servers, A, B and C. One majority is A and B. Another is B and C. Both include B. Any two majorities always share at least one server, so the majority that picks a new leader always includes a server that saw every confirmed write. Three members survive one failure, five survive two. Kafka uses this same idea for its controllers (KRaft), but for the data itself it uses the ISR, which you tune.

RabbitMQ's older mirrored queues (classic queues copied to other servers) are gone. They were removed in RabbitMQ 4.0, after three years of warnings, and quorum queues are the default choice for a replicated queue.

Keeping history, and reading it again

A Kafka message is deleted by a rule about age or size, never because someone read it. The default is 7 days and no size limit. So readers can sit at very different places: a live service at the end, a nightly job a day behind, a new service reading from the start.

This changes how you fix mistakes. Ship a bug that handled yesterday's events wrongly, fix it, move the group's offset back, and read the events again. The cost is disk: days of a busy topic take a lot of space. Tiered storage, which moves old parts of the log to cheaper storage, has been production-ready since Kafka 3.9. Kafka itself ships no production plugin for the remote store, and compacted topics are not supported there.

Log compaction is the other kind of retention. Kafka keeps at least the last value for each key and cleans out older ones. A compacted topic is like a table of the current state that you can always read from the start. Kafka's own __consumer_offsets works this way, and so do most 'rebuild the cache from Kafka' setups.

RabbitMQ streams keep a log too, and readers can start anywhere with x-stream-offset. One warning from the docs: a stream has no retention limit by default, so it grows until the disk is full unless you set max-age or max-length-bytes.

Why Kafka can move more messages

Kafka is fast because it does very little work for each message. RabbitMQ does more work for each one: the routing decision, the per-message ack, the queue's state. That work gives flexibility, and it takes time.

Four design choices help Kafka. Writes only add to the end of a file. Writing in one long run like this is much faster than jumping around the disk. Producers send messages in batches, often compressed, so there are fewer requests. Kafka leans on the operating system's page cache (memory the OS uses to hold recently used file data) instead of its own cache. And for plain connections it uses sendfile, a 'zero-copy' call that moves bytes from that cache to the network without copying them into Kafka's memory. The docs add a catch: with TLS (encryption) turned on, sendfile is not used, because encryption happens in normal program memory.

How big is the difference in throughput (messages per second)? Neither project publishes a fair side-by-side speed test, and we did not run one: a speed test on one laptop does not show how a real cluster behaves. RabbitMQ's streams page explains why it built streams: 'No persistent queue types are able to deliver throughput that can compete with any of the existing log based messaging systems.' That sentence is about RabbitMQ's own older queue types. Its acknowledgements page adds that if throughput matters most, you should use streams, not a bigger prefetch.

Push, pull and the slow reader

RabbitMQ pushes. A worker subscribes (basic.consume, which the docs recommend over polling with basic.get), and the broker sends messages as they arrive, up to the prefetch limit. When a worker holds too many unacked messages, the broker stops sending to it. A slow worker just gets less. If no worker keeps up, the queue grows, and TTL and dead-letter rules start to act.

Kafka readers pull. A reader calls poll() in a loop and fetches a batch when it is ready, with a 'long poll': the request waits for new data to arrive instead of asking again and again. A slow reader simply falls behind. The gap between the end of the log and its bookmark is called consumer lag, and it is the first number to watch on any Kafka system. There are two limits. If a reader falls further behind than the retention time, its unread messages are deleted before it gets to them. With the default auto.offset.reset=latest, it then jumps to the newest message without an error. And a reader far behind reads old data from disk, which adds load on the same brokers that serve everyone else.

Kafka's model has a weak spot. In a normal consumer group there is no ack for each message and no built-in dead-letter queue (Kafka Streams apps, a separate library, got one in 4.2). A poison message, one that makes the reader fail every time, stops every message behind it in that partition until your code skips it or puts it aside. RabbitMQ's per-message acks, rejects, requeues and dead-letter exchanges exist to solve exactly that, and quorum queues stop redelivering a message after 20 tries by default. Set a dead-letter exchange, because without it that message is deleted. Dead-lettering is at most once by default. For messages you cannot lose, set dead-letter-strategy=at-least-once and also overflow=reject-publish, or RabbitMQ quietly falls back to at most once. Note that in 4.3 a nack with requeue does not count toward the limit; a reject or a crash does. Kafka's share groups now bring per-message acks and a delivery count to Kafka too.

What each one costs to run

Both take work to run, and the person on call (the engineer who must fix problems at night) feels it.

Kafka used to need a second system, ZooKeeper, to keep track of the cluster. Kafka 4.0 (March 2025) removed it. Kafka now runs in KRaft mode: a small Raft group of controller servers, usually 3 or 5, keeps the cluster's metadata (which servers, topics and partitions exist), and a majority of them must be up. Teams still on ZooKeeper must first move to KRaft on a 3.x version (the project recommends 3.9), then upgrade to 4.x. You still plan partitions, disks, retention and consumer lag. Adding a broker does not move existing partitions to it; you move them yourself (partition reassignment). And you cannot reduce the number of partitions of a topic.

RabbitMQ is quick to start on one server and pleasant for task queues. Clusters need more care. When a node runs short of memory or disk, RabbitMQ raises an alarm and blocks every connection that publishes, not only the busy queue. Its queues work best when they stay short. Each queue has one leader on one server, so to go faster you spread work over many queues or use super streams (partitioned streams, since 3.11). Since RabbitMQ 4.3, the cluster's metadata lives only in Khepri, a store built on Raft, so a cluster needs more than half of its servers online to keep working. Quorum queues also need a majority of their members.

Managed services, where a cloud company runs the brokers for you (for example Amazon MSK for Kafka, or Amazon MQ for RabbitMQ), take much of this work away, and you pay for it. When someone says one is 'simpler', ask: simpler for whom, and at what size?

What changed in 2025 and 2026

Many comparisons online are a few years old. These are the changes that matter, each from the projects' own release notes.

Kafka 4.0 (March 2025): no ZooKeeper at all, KRaft only, and the new rebalance protocol became generally available. Kafka 4.2 (February 2026): 'Kafka Queues (Share Groups) is now production-ready'. Kafka 4.3 (May 2026): the start of retiring the classic rebalance protocol in the consumer. Kafka 4.4 is not out yet; only release candidates exist.

RabbitMQ 4.0 (September 2024): classic mirrored queues removed, AMQP 1.0 built in, and quorum queues stop redelivering after 20 tries by default. RabbitMQ 4.2 (October 2025): Khepri is the default metadata store for new clusters, and streams gained SQL filter expressions. RabbitMQ 4.3 (April 2026): Mnesia removed, so Khepri is the only store, and quorum queues got strict priorities and delayed retries.

So each product now does a little of the other's job. Kafka can act like a work queue, and RabbitMQ streams act like a log. The main design has not changed: Kafka is still a log first, and RabbitMQ is still a router of queues first.