Competing Consumers

One cashier, fifty customers in line. The wait is brutal. So the store opens five more registers. Now six cashiers compete to serve the next customer. Each customer goes to whichever cashier is free. The line drains six times faster.

Competing consumers is the messaging equivalent. Instead of one consumer processing all messages from a queue, you run multiple consumer instances. The broker distributes messages among them, each message goes to exactly one consumer. The queue drains faster because work is parallelized across multiple workers.

This is the fundamental scaling pattern for message processing. When a single consumer can't keep up with the message rate, you don't optimize the consumer. You add more consumers. Kafka consumer groups, RabbitMQ's round-robin dispatch, SQS with multiple pollers: they all implement competing consumers.

The strength is in the simplicity. The producers don't change. The broker doesn't change. The message format doesn't change. You just add more consumer instances, and throughput increases proportionally (up to the broker's parallelism limit). LinkedIn processes over 7 trillion messages per day through Kafka consumer groups. Uber routes hundreds of millions of ride events through competing consumer pools. The pattern scales from your side project to planet-scale infrastructure.

The broker maintains the queue and distributes messages to registered consumers. Each message is delivered to exactly one consumer in the group.

Distribution strategies vary by broker, and the choice matters more than most teams realize.

RabbitMQ defaults to round-robin: messages go to Consumer 1, then Consumer 2, then Consumer 3, then back to 1. Simple, but it ignores processing speed entirely. If Consumer A takes 10 seconds per message and Consumer B takes 100ms, they still get the same number of messages. Consumer B finishes and idles while Consumer A's backlog grows.

Kafka takes a different approach with consumer groups. Each partition in a topic is assigned to exactly one consumer in the group. If you have 12 partitions and 4 consumers, each consumer reads from 3 partitions. If a consumer crashes, its partitions are reassigned to the remaining consumers (a process called rebalancing). The partition count is your parallelism ceiling. You can never have more active consumers than partitions.

SQS uses pull-based distribution. Consumers poll the queue when they're ready. No assignment, no round-robin. Fast consumers poll more often and naturally get more messages. The queue handles backpressure automatically.

Prefetch and fairness. RabbitMQ's default round-robin distributes messages evenly by count, not by processing time. Setting prefetch_count = 1 fixes this. Each consumer gets one message at a time and only receives the next when it ACKs the current one. Slower consumers naturally get fewer messages. RabbitMQ's documentation recommends prefetch between 1 and 30 depending on workload variance. Shopify's engineering team found that tuning prefetch from 25 to 1 on their order processing queue reduced p99 latency by 40% because slow messages stopped blocking fast consumers.