Load Balancing Proxies

Multicast

0 of 10 complete

0%

Contents

Back|Load Balancing ProxiesMulticast
1/10
27 min left
  1. Home
  2. System Design: Foundation
  3. Load Balancing & Proxies
  4. Multicast
Prerequisites
Broadcastrequired
Related Topics
Ingress ControllersKubernetes & ContainersLatencyCore FundamentalsBandwidthCore FundamentalsConnection TimeoutCore FundamentalsRequest TimeoutCore Fundamentals
Previous lessonBroadcastNext lessonAnycast Routing

System Design

1 of 10
  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Privacy
  • Terms

The Best Idea in Networking That Nobody Deployed

Unicast sends to one machine and scales badly when many want the same thing. Broadcast reaches everybody and wastes the time of everybody who does not care.

Multicast is the obvious answer to both, and it is elegant. A sender transmits one copy. Machines that want it join a group. The network duplicates the packet only where the paths to two members diverge, and nowhere else.

A tree diagram. The sender sends one copy to a router, which forwards to two more routers. Each of those delivers to two members. One node beside them is marked not a member and gets nothing.
FlowchartFigure 1. The sender sends one copy, and the network makes the rest

Each router copies a packet only when it has members down more than one interface. A link carries at most one copy of the stream, however many members are beyond it.

A machine that has not joined receives nothing at all, which is exactly what separates this from broadcast.

A line chart of gigabits per second at the sender against interested receivers in thousands. The unicast line rises straight to 50 gigabits at ten thousand receivers. The multicast or broadcast line is flat at effectively zero. Notes read 5 Mbps each sent separately, and 5 Mbps once.
ChartFigure 2. Ten thousand viewers costs the sender the same as one

This is arithmetic rather than a benchmark, and the flat line is the whole argument for the protocol.

The difference between multicast and broadcast is not visible on this chart, because it is not about the sender. It is about who else pays.

So multicast gives you the profile of broadcast with the targeting of unicast. It has worked for thirty years. It carries an enormous amount of real traffic today.

And it is essentially absent from the public internet, which is the more interesting half of this lesson.

A Group Address Is Not a Machine

The whole of 224.0.0.0/4 is multicast, which is one sixteenth of the IPv4 address space set aside for this. A group address does not belong to any device. It names a conversation that devices can join.

Where in that range your group sits changes what happens to your packets.

A table of four multicast ranges. 224.0.0.0/24 is link-local control and is never forwarded by a router, ever. 224.0.1.0 to 238.x is globally assigned, registered with IANA for internet-wide use. 232.0.0.0/8 is source-specific, where you join a source and group pair rather than just a group. 239.0.0.0/8 is administratively scoped: private, like 10.0.0.0/8 but for groups.
FigureFigure 3. Four ranges with four different sets of rules

224.0.0.0/24 is the one worth memorising. Routers must never forward it, whatever the hop limit says, and that is a rule in the router rather than a consequence of .

That is where OSPF, IGMP and mDNS live, because all three are conversations between neighbours on one wire.

An isometric diagram of three nested square slabs of increasing size. The smallest is labelled 224.0.0.0/24, this wire only. The middle is 239.0.0.0/8, this organisation. The largest is 224.0.1.0 to 238, the whole internet in theory. A line beneath reads that the range you pick decides how far your packets are allowed to travel.
Isometric

How the Network Knows Who Wants It

Every question in multicast reduces to one: how does each router know whether it has any members down each of its interfaces?

On a single subnet: IGMP

A sequence diagram with three lifelines: a host, the router acting as group querier, and other hosts. The host joins 239.1.1.1 the moment the application asks. The router sends a periodic query asking whether anyone still wants anything, roughly every two minutes. One member replies for the whole group and the others stay quiet to avoid a flood of reports. The host later leaves, and the router sends a group-specific query asking whether anyone else wants it, then stops forwarding after a second or two of silence.
SequenceFigure 5. IGMP is a router repeatedly asking whether anybody still cares

Membership is soft state: it expires unless somebody keeps confirming it.

That is the right design here. If a host crashes it never sends a leave, and hard state would never be cleaned up. Expiring by default means a dead member costs you one query interval and nothing more.

And the switch has to care too

A sequence diagram with three lifelines: a sender, the switch, and ports with no member. One multicast frame arrives and is flooded to every port, noted as a switch having no idea what a group is because it sees an unknown MAC. With snooping the switch watches the IGMP messages go past. The next multicast frame is not sent to those ports at all, only to the ones that asked.
SequenceFigure 6. Without IGMP snooping, multicast on a switch is just broadcast

Where the Model Meets the Cable

Everything above happens at the IP layer. A network card cannot filter on IP addresses at all, so there is one more step that is rarely taught and produces a genuinely surprising consequence.

A hand-drawn diagram. A group address has 28 significant bits: 224.0.0.251 shown in binary. An arrow labelled keep only the low 23 leads to the result: prefix 01:00:5e plus 00:00:fb equals 01:00:5e:00:00:fb. A handwritten note reads that 28 bits in and 23 bits out means 32 different groups land on the same hardware address.
SketchFigure 9. The group becomes a MAC, and five bits are thrown away

The mapping is arithmetic with no lookup: a fixed prefix, then the low 23 bits of the group address. Twenty-eight bits do not fit in twenty-three.

So 32 distinct group addresses arrive at the card as the same hardware address.

Neither the joins nor the arithmetic are theoretical. Both are printable on an ordinary laptop that nobody has configured for multicast.

A recorded terminal session. netstat -g shows the multicast groups this machine has joined on en0: 01:00:5e:00:00:01 which is 224.0.0.1 all hosts on this subnet, 01:00:5e:00:00:fb which is 224.0.0.251 mDNS for Bonjour AirPlay and Chromecast, and four IPv6 groups with the 33:33 prefix. The session then works through the mapping: the low 23 bits of 224.0.0.251 are 00:00:fb, so prefix 01:00:5e plus that gives 01:00:5e:00:00:fb. Finally four group addresses, 224.0.0.251, 225.0.0.251, 231.128.0.251 and 239.128.0.251, are all shown mapping to the same hardware address.
RecordedFigure 10. Two IPv4 groups and four IPv6 ones, joined by software that came with the machine

Why It Is Not on the Internet

A protocol this efficient not being used everywhere needs an explanation, and "ISPs found it complex" is only part of it.

Five reasons. Every router must participate, so one uncooperative network in the middle breaks the tree and nobody controls the middle. State per group per router, because unicast routers hold routes while multicast routers hold a tree per active group. No way to charge for it, since the sender pays for one stream and the transit networks carry all the copies. No congestion control, being UDP with no feedback loop. And security is unsolved, because anyone can send to a group and the network delivers it to every member.
FigureFigure 11. Five reasons, and only the first one is technical

The technical requirement is that every router on every path has to cooperate. The business requirement is harder: transit networks carry copies that nobody pays them for.

Without congestion control a multicast stream cannot be a good citizen on a shared link, and a group has no access control, so anybody can inject traffic into somebody else's stream.

Three of those deserve to be seen rather than listed.

An isometric diagram of three stacks of slabs of increasing height, labelled unicast with one entry per prefix at two slabs, multicast ASM with one tree per group at five slabs, and multicast SSM with one tree per source and group at nine slabs. Text beneath reads that unicast state is bounded by the address space while multicast state is bounded by how many streams exist.
IsometricFigure 12. A unicast router remembers places. A multicast router remembers conversations.

The Model That Fixes Most of This

Almost every problem above comes from one design decision made in the 1980s: you join a group, and anybody may send to it.

Two panels. Any-source ASM, join star comma G: you subscribe to a group and receive from anybody sending to it, the network must find the senders which needs a rendezvous point, anyone can inject traffic into your group, and this is the original model and the reason the routing is complicated. Source-specific SSM, join S comma G: you subscribe to a group from one named source and nothing else, no rendezvous point is needed so the tree is built straight to that source, traffic from anybody else is discarded, and this is 232.0.0.0/8 which is what large deployments actually use.
FigureFigure 15. The difference between the two models is one word in the subscription

Under any-source multicast the network has to discover who is sending, and that discovery is what rendezvous points and shared trees exist to do. It is most of the complexity in the protocol.

Under source-specific multicast the receiver already names the source, so the tree is built directly to it and unnamed senders are discarded.

If you are designing anything with multicast today, use source-specific. Any-source is the model that made inter-domain multicast intractable, and both the routing complexity and the injection problem disappear the moment the receiver names the sender.

What Won Instead

Four approaches with logos. CDN edge caching sends one copy per edge and then unicast to each viewer nearby. Adaptive streaming uses HTTP chunks cached like any other file, needing no special network. A WebRTC SFU forwards one incoming stream to many, in software. Peer to peer makes receivers into senders, which is the other way to avoid the fan-out. A note reads that none of them needs a single router anywhere to know anything about groups, which is exactly why they were deployable.
ArchitectureFigure 16. The industry solved one-to-many by moving the sender, not by changing the network

Every approach here replicates in software, at a place that somebody owns and can bill for.

That is the deployability lesson, and it generalises well beyond networking: a protocol that asks independent parties to cooperate for somebody else's benefit is very hard to ship.

The concept survived even though the protocol did not. Send once, deliver to interested parties, is exactly what a message broker's topic is, what a WebRTC forwarding unit does, and what a edge does for a live stream. All of it in software, none of it in routers.

But it is very much alive inside walls

Four places multicast carries real traffic. Financial market data: one exchange, one data centre, thousands of subscribers to the same feed, where microseconds matter and a fan-out server would add some. IPTV over a carrier network: one operator owns every router between the head end and the set-top box, and a live channel is genuinely the same bytes at the same moment. Routing protocols: OSPF, RIPv2 and IGMP itself, all on the link-local range, neighbours on one wire where the alternative is broadcast. Service discovery: mDNS and SSDP on every network you own, the modern replacement for LAN broadcast.

IPv6 Made It Compulsory

The previous lesson covered IPv6 removing broadcast entirely. The other half of that decision is that multicast stopped being optional.

Four well-known IPv6 groups. ff02 colon colon 1 is all nodes on this link, the replacement for the broadcast address. ff02 colon colon 2 is all routers on this link, how a host finds its default gateway. ff02 colon colon fb is mDNS, the same discovery you use every day over IPv6. ff02 colon colon 1 colon ff00 colon 0 slash 104 is solicited-node, derived from the last 24 bits of the target's address. A note explains that instead of asking every host who owns an address, IPv6 asks a group derived from the address itself, so typically exactly one machine is disturbed.
FigureFigure 19. IPv6 cannot function without multicast, by design

There is no broadcast address to fall back on, so the basics of joining a network run on groups.

The solicited-node group is the clever one. Because it is derived from the target's own address, an address lookup disturbs roughly one machine instead of all of them.

That is a design pattern worth taking away. Broadcast asks everybody a question meant for one machine. Deriving a narrow group from the question itself gets the same reach at almost none of the cost, and it needed no new mechanism because multicast already existed.

When to Reach for It

A decision flowchart. Same bytes, same moment, to many receivers? If no, unicast. If yes, do you control every router on the path? If no, use a CDN or a fan-out server, because multicast will not survive the first network you do not own. If yes, can the receivers tolerate loss? If no, multicast plus a repair mechanism or a second copy on a separate path. If yes, multicast, source-specific, in 232.0.0.0/8 or 239.0.0.0/8.
FlowchartFigure 20. Three questions, and the second one eliminates almost everybody

Ownership of the path is the gate, and everything else is detail.

If loss is unacceptable you need a repair mechanism, and that is real engineering rather than a configuration flag.

In an interview, the second box is the answer worth giving. "Multicast, if this is all inside our own network, otherwise a or a fan-out service" shows that you know why the technically better option is usually unavailable, which is a more useful thing to know than the protocol itself.

One last piece of pedantry that is worth being pedantic about. People say "we multicast the events to all the consumers" when they mean a broker delivers a topic to several subscribers. That is application-level fan-out over unicast connections. It is a fine design and it is not multicast, and confusing the two leads to expecting properties you are not going to get.

Quick Check

Knowledge Check

6 questions - Score 80% to pass

Q1

A server streams live video to 5,000 viewers on a corporate network. Which uses the least server bandwidth?

Q2

Your multicast video stream is working, but the network team reports that it is reaching every port on one particular switch, including machines that never joined. What is the most likely cause?

Q3

Why do 32 different multicast group addresses arrive at a network card as the same hardware address?

Q4

Why is multicast essentially absent from the public internet?

Q5

You are designing a market data feed inside a data centre you own. Which model should you choose and why?

Q6

A colleague says 'Kafka multicasts events to all the consumers in the group.' What is wrong with that sentence?

Figure 4. How far a packet travels is decided by which range you chose

This is genuinely different from unicast, where any address can in principle be reached from anywhere and the boundaries are policy. Here the scope is carried in the address itself.

Putting a neighbour-to-neighbour protocol in a range that physically cannot leave the wire is stronger than any firewall rule.

If you are choosing a group address for something inside your own network, use 239.0.0.0/8. Picking something in the globally assigned range because it looked unused is the multicast equivalent of squatting on somebody else's public IP block.

A switch works on hardware addresses and has no concept of a group. A multicast frame's destination MAC is not one it has ever learned a port for, so the default behaviour for an unknown destination applies: flood it everywhere.

Snooping means the switch reads the join and leave messages passing through it and builds its own table.

This is the single commonest reason a multicast deployment behaves like broadcast and nobody can work out why. The routing is right, the joins are right, and one switch in the middle has snooping turned off and is flooding a video stream to every port on it.

Between subnets: PIM

A sequence diagram with three lifelines: a new member, its router, and the router upstream toward the source. The member sends an IGMP join for a source and group. Its router sends a PIM join toward the source, using the ordinary unicast route to that source, backwards. The upstream router adds this interface to the tree for that source and group, and passes the join further up if it was not already a member. The stream then flows down.
SequenceFigure 7. Nobody plans the tree. Each receiver grafts itself on.

PIM stands for protocol-independent multicast, and the name is the explanation: it does not compute routes of its own. It looks the source up in the ordinary unicast table and sends its join in that direction.

The tree is therefore built branch by branch from every leaf, and pruned the same way.

A hand-drawn diagram. A packet from source S arrives on interface B. The unicast table says S is reached via interface A. So this packet arrived the wrong way and is dropped. Handwritten notes read that reverse path forwarding means accepting a packet only on the interface you would use to send to its source, which makes a loop impossible.
SketchFigure 8. Accept it only if it came in the direction you would send back

Reverse path forwarding is the entire loop-prevention mechanism and it is one lookup. A multicast packet can arrive from more than one direction, and the router discards it unless it came in on the interface it would use to reach that source.

That single check makes a forwarding loop structurally impossible.

It also means multicast depends on the unicast routing being symmetric enough. On a path where the route to the source differs from the route the traffic takes, RPF discards perfectly good packets, and that is one of the classic reasons a deployment fails mysteriously.

Nobody configured this. The operating system's discovery services joined those groups on their own, which is worth knowing before you conclude that multicast is something other people run.

The last block is the collision, and it is measurable rather than theoretical: the card accepts traffic for all four of those groups, and the kernel discards the three you never joined, in software, on every packet.

Choosing group addresses that differ only in those five discarded bits is therefore a real way to make a host slower for no benefit.

A unicast forwarding entry covers a whole block of addresses and is shared by everybody using it. A multicast entry exists for one group and disappears when the last member leaves.

So the table's size tracks how many streams are running, not how big the internet is.

That is fine on a network where you know how many streams there are and impossible on one where you do not. A transit provider cannot bound the state, cannot predict it, and gains nothing from carrying it.

A sequence diagram with three lifelines: the sender with one stream to many members, member A, and member B on a congested link. Packets 1 to 500 reach A. Packet 312 is lost on the way to B. B asks the sender to resend 312, noted as every member doing this putting the sender back to unicast. Resending 312 to everybody wastes it on A.
SequenceFigure 13. There is no acknowledgement, because acknowledgements do not scale

works because there are exactly two ends. Multicast has one sender and an unknown number of receivers, and every reliability mechanism you might add breaks something.

Acknowledge everything and the sender drowns. Retransmit to the group and everybody pays for one member's loss. Retransmit by unicast and the fan-out you were avoiding has returned.

A line chart of percent of the link against seconds. A TCP flow climbs, drops sharply, climbs again and drops, in a sawtooth between roughly 20 and 95 percent. A multicast stream is a flat line at 55 percent for the whole 20 seconds. A dashed rule at 100 percent is labelled the link is full above this.
ChartFigure 14. One of these two flows is being a good citizen, and it is not the flat one

TCP reads loss as congestion, halves its rate and climbs again. That sawtooth is what lets many flows share a link without any of them collapsing.

A multicast stream has no feedback loop, so it cannot know the link is full. It keeps sending at its configured rate and the TCP flows around it back off instead.

So an uncontrolled multicast stream does not merely fail to share, it takes from every well-behaved flow beside it. That is why carriers admit multicast only onto capacity they have planned for, and a large part of why nobody wants it arriving from outside.

Figure
Figure 17. Four places multicast carries real traffic every day

Read the middle column of every row and the pattern is identical: one organisation controls every router on the path.

The common property is ownership rather than scale. Inside that boundary multicast is excellent and it is not going anywhere.

Four rows. Bonjour and AirPlay use 224.0.0.251 and ff02 colon colon fb, mDNS, on every network you use. Some Kubernetes CNI plugins use 239.0.0.0/8 internally for node discovery where the operator allows it. Not Kafka: unicast, always, listed as a common mistaken assumption worth stating. Not a CDN either: unicast from the edge, because the fan-out is a server rather than the network.
ArchitectureFigure 18. You use multicast every day and almost never on purpose

The two rows marked "not" are deliberate. Assuming a system uses multicast because it delivers to many consumers is a common and occasionally expensive mistake.

Multicast means the network duplicates the packet. If a server is looping over a list of recipients, that is fan-out, it is unicast, and it has none of multicast's properties or problems.