We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Messaging & Event Systems
A buffer that stores messages between producers and consumers. Messages are processed one by one, in order. Think of it as a to-do list for your services.
In short
A message queue is a component that stores messages sent by producers until consumers are ready to process them, decoupling the two so they never have to run at the same time or the same speed. Producers add messages to the queue and return immediately; consumers pull messages off the queue one at a time, process them, and acknowledge completion.
In short
A message queue is a component that stores messages sent by producers until consumers are ready to process them, decoupling the two so they never have to run at the same time or the same speed. Producers add messages to the queue and return immediately; consumers pull messages off the queue one at a time, process them, and acknowledge completion.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Message Queue across a run of lessons, not one page. These 3 alone are about 71 minutes of step-by-step reading, every one with a quiz.
Decouple producers from consumers with a buffer that holds messages until they're processed.
Design a multi-channel notification system - push, email, SMS with priority queues, fan-out, and delivery guarantees.
Testing how components work together, databases, APIs, message queues, and third-party services in combination.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Message Queuespay once,
yours for life
A message queue sits between two parts of a system. One side, the producer, creates a unit of work and drops it into the queue. The other side, the consumer, takes work out of the queue and handles it. The queue itself is just durable storage with ordering rules, but that small piece changes how the whole system behaves.
The point is decoupling. Without a queue, when a user uploads a video your web server has to transcode it right then while the user waits. With a queue, the web server writes a short message like transcode video 8842 and responds in 20 milliseconds. A separate pool of workers picks up that message later and does the slow transcoding job. The producer and consumer never call each other directly and never have to be online at the same moment.
Most queues give you delivery guarantees. A message is not removed for good until the consumer sends an acknowledgement saying it finished. If the worker crashes mid-job, the message reappears and another worker retries it. That is why queues are the standard tool for work that must not be lost.
A producer connects to a broker such as RabbitMQ or Amazon SQS and publishes a message, usually a small JSON or binary blob with the data needed to do the job. The broker writes it to storage, often to disk so it survives a restart, and the producer is done.
Consumers either poll the queue asking are there messages, or hold an open connection and have messages pushed to them. When a consumer takes a message, the broker hides it from other consumers for a visibility timeout, commonly 30 seconds. If the consumer acknowledges within that window the message is deleted. If it does not, the broker assumes the consumer died and makes the message available again.
Two failure modes matter. At-least-once delivery means a message can be delivered more than once if an acknowledgement is lost, so consumers must be idempotent, meaning processing the same message twice does no harm. Messages that fail repeatedly are moved to a dead letter queue after a retry limit, usually 3 to 5 attempts, so one poison message does not block the rest. Scaling up is simple: add more consumer instances and they share the load because each message goes to exactly one of them.
Reach for a queue when work is slow, spiky, or must survive failure. Sending email, generating PDFs, charging cards, resizing images, and syncing data to another system are classic cases. A queue lets you absorb a traffic spike by letting the backlog grow instead of crashing, and lets you scale producers and consumers independently.
The cost is added latency and complexity. A request that goes through a queue is no longer instant, so a message queue is the wrong tool when you need an answer in the same request, like a database lookup feeding a page render. You also take on operational work: monitoring queue depth, tuning visibility timeouts, handling duplicate delivery, and watching the dead letter queue.
A point-to-point message queue is also different from publish subscribe. In a queue each message is consumed by one worker, which is right for task distribution. If many independent services each need a copy of every event, you want a pub sub topic or a log like Kafka instead. Many systems use both.
Picture an online store on Black Friday. A customer clicks buy. The checkout service charges the card, then needs to send a confirmation email, update inventory, notify the warehouse, and refresh analytics. Doing all of that inline would make checkout slow and fragile, because if the email provider is down the whole purchase could fail.
Instead the checkout service writes one message per task to a queue and returns success to the customer in well under 100 milliseconds. Background workers drain the queue: the email worker sends the receipt, the inventory worker decrements stock, the warehouse worker creates a pick ticket. If the email service has an outage, those messages simply wait in the queue and get retried when it recovers. The customer's order was never at risk, and during the spike the queue depth grows to tens of thousands and drains back down once traffic falls.
Amazon SQS
Fully managed queue used across AWS to decouple microservices and buffer work, handling trillions of messages a month.
RabbitMQ
Popular open source broker that implements traditional point-to-point queues and flexible routing for task distribution.
Redis
Used as a lightweight queue through lists or Redis Streams, powering job systems like Sidekiq, Celery, and Bull.
Uber
Queues ride requests, payment events, and notifications so background services process them without blocking the rider app.
A database is for storing and querying data you read back many times. A queue is transient storage for work in transit: messages are added, processed once, and removed. You can build a crude queue on a database, but dedicated queues handle ordering, visibility timeouts, retries, and at-scale throughput far better.
A classic message queue delivers each message to one consumer and deletes it after acknowledgement, which is ideal for distributing tasks. Kafka is a durable log where messages are retained and many independent consumer groups can each read the full stream. Use a queue for work distribution and Kafka for event streaming and replay.
The message is not deleted until the consumer acknowledges it. If the consumer crashes before acknowledging, the broker's visibility timeout expires and the message becomes available again for another consumer to retry. This is why queues are reliable for critical work.
Most queues guarantee at-least-once delivery, so a message can occasionally be delivered twice, for example if an acknowledgement is lost. Idempotent means processing the same message twice produces the same result, so a duplicate charge or duplicate email is avoided. Tracking a unique message id is a common way to achieve this.
It is a separate queue where messages go after they fail processing too many times, usually 3 to 5 attempts. This stops one broken message from blocking everything behind it and gives you a place to inspect and reprocess failures later.
Pub/Sub
A messaging pattern where publishers send messages to topics, and subscribers receive messages from topics they care about. Publishers don't know who's listening.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Microservices
An architecture where an application is split into small, independent services that communicate over the network. Each service owns its own data and can be deployed separately.
Webhook
An HTTP callback triggered by an event. Instead of polling for updates, the source system pushes a notification to your URL when something happens.
Kafka Topic
A named feed or category to which producers publish records. Topics are split into partitions for parallelism, and each partition is an ordered, immutable log.
Kafka Partition
A subset of a Kafka topic that provides ordering guarantees and parallel processing. Each partition lives on one broker and can be consumed by one consumer per group.
This page explains the idea. The "Message Queues" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Message Queues lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Message Queue in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
