Message Queue

A message queue sits between two parts of a system. One side, the producer, creates a unit of work and drops it into the queue. The other side, the consumer, takes work out of the queue and handles it. The queue itself is just durable storage with ordering rules, but that small piece changes how the whole system behaves.

The point is decoupling. Without a queue, when a user uploads a video your web server has to transcode it right then while the user waits. With a queue, the web server writes a short message like transcode video 8842 and responds in 20 milliseconds. A separate pool of workers picks up that message later and does the slow transcoding job. The producer and consumer never call each other directly and never have to be online at the same moment.

Most queues give you delivery guarantees. A message is not removed for good until the consumer sends an acknowledgement saying it finished. If the worker crashes mid-job, the message reappears and another worker retries it. That is why queues are the standard tool for work that must not be lost.

A producer connects to a broker such as RabbitMQ or Amazon SQS and publishes a message, usually a small JSON or binary blob with the data needed to do the job. The broker writes it to storage, often to disk so it survives a restart, and the producer is done.

Consumers either poll the queue asking are there messages, or hold an open connection and have messages pushed to them. When a consumer takes a message, the broker hides it from other consumers for a visibility timeout, commonly 30 seconds. If the consumer acknowledges within that window the message is deleted. If it does not, the broker assumes the consumer died and makes the message available again.

Two failure modes matter. At-least-once delivery means a message can be delivered more than once if an acknowledgement is lost, so consumers must be idempotent, meaning processing the same message twice does no harm. Messages that fail repeatedly are moved to a dead letter queue after a retry limit, usually 3 to 5 attempts, so one poison message does not block the rest. Scaling up is simple: add more consumer instances and they share the load because each message goes to exactly one of them.

Reach for a queue when work is slow, spiky, or must survive failure. Sending email, generating PDFs, charging cards, resizing images, and syncing data to another system are classic cases. A queue lets you absorb a traffic spike by letting the backlog grow instead of crashing, and lets you scale producers and consumers independently.

The cost is added latency and complexity. A request that goes through a queue is no longer instant, so a message queue is the wrong tool when you need an answer in the same request, like a database lookup feeding a page render. You also take on operational work: monitoring queue depth, tuning visibility timeouts, handling duplicate delivery, and watching the dead letter queue.

A point-to-point message queue is also different from publish subscribe. In a queue each message is consumed by one worker, which is right for task distribution. If many independent services each need a copy of every event, you want a pub sub topic or a log like Kafka instead. Many systems use both.

Picture an online store on Black Friday. A customer clicks buy. The checkout service charges the card, then needs to send a confirmation email, update inventory, notify the warehouse, and refresh analytics. Doing all of that inline would make checkout slow and fragile, because if the email provider is down the whole purchase could fail.

Instead the checkout service writes one message per task to a queue and returns success to the customer in well under 100 milliseconds. Background workers drain the queue: the email worker sends the receipt, the inventory worker decrements stock, the warehouse worker creates a pick ticket. If the email service has an outage, those messages simply wait in the queue and get retried when it recovers. The customer's order was never at risk, and during the spike the queue depth grows to tens of thousands and drains back down once traffic falls.