Load Balancing Proxies

Broadcast

0 of 9 complete

0%

Contents

Back|Load Balancing ProxiesBroadcast
1/9
27 min left
  1. Home
  2. System Design: Foundation
  3. Load Balancing & Proxies
  4. Broadcast
Prerequisites
Unicast Routingrequired
Related Topics
Ingress ControllersKubernetes & ContainersLatencyCore FundamentalsBandwidthCore FundamentalsConnection TimeoutCore FundamentalsRequest TimeoutCore Fundamentals
Previous lessonUnicast RoutingNext lessonMulticast

System Design

1 of 9
  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Privacy
  • Terms

The Chicken and Egg Problem at the Bottom of Every Network

Here is a problem with no obvious solution. A laptop joins a network. To send a message to anybody, it needs their address. To get an address of its own, it needs to ask a server. To ask the server, it needs the server's address.

It has none of these. It has not been told anything at all.

Broadcast is the answer, and it is the only answer: send a message addressed to everybody, on the assumption that whoever can help will notice. Every machine on the local network receives it, almost all of them ignore it, and one of them replies.

That sounds wasteful and it is. It is also the mechanism holding up the bottom of every network you have ever used, and it is worth understanding properly rather than as an analogy, because two of the failures it causes look identical and are not.

An isometric diagram of six machine blocks standing on a base slab, with a taller router block at the right edge labelled the router and the wall. Text reads: one frame sent, six machines interrupted, every one of them has to look at it before deciding to ignore it. A line beneath reads that the router does not forward it, and that boundary is the whole of the design.
IsometricFigure 1. Every machine receives it, and every machine has to look

The cost of broadcast is not the . One small frame is nothing. The cost is that every machine's network card accepts it and every machine's kernel has to decide what to do with it.

That is a tiny interrupt per host per frame, which is nothing at all until there are several thousand hosts.

Where the Broadcast Address Comes From

You will often see "the broadcast address is the last address in the subnet". That is true, and it is a rule to memorise rather than something to understand. It is actually arithmetic, and doing it once removes the need to remember it ever again.

Four rows showing binary arithmetic. 192.168.0.124, the machine, in binary. 255.255.255.0, the mask slash 24, in binary. Then 192.168.0.0, the result of ANDing them, described as the network with all host bits zero. Then 192.168.0.255, the result of ORing with the inverted mask, described as the broadcast with all host bits one. A note explains that the mask marks the boundary and that setting all host bits to one is the broadcast address by definition rather than by convention.
FigureFigure 2. The broadcast address is the network with every host bit set to one

The mask says how many leading bits are the network part. AND the address with the mask and the host bits go to zero, which is the network itself. OR it with the inverted mask and the host bits go to one, which is the broadcast.

Both of those addresses are reserved, which is why a /24 holds 254 usable hosts rather than 256.

That arithmetic is not a claim. It can be checked on any machine in about twenty seconds.

A recorded terminal session. ifconfig reports inet 192.168.0.124, netmask 0xffffff00, broadcast 192.168.0.255. The mask converts to 255.255.255.0 which is a slash 24. Address AND mask gives 192.168.0.0, the network itself, not usable by a host. Address OR inverted mask gives 192.168.0.255, the broadcast address, and ifconfig agrees. Usable host addresses are 2 to the power of 8 minus 2, which is 254, minus 2 because the all-zeros and all-ones addresses are spoken for.
RecordedFigure 3. Computed from a real interface, and ifconfig agrees

Everything below the first line is derived from the two numbers on it.

The last part is where this matters in practice. A /30 leaves two usable addresses, which is exactly a point-to-point link between two routers. A /31 leaves none at all.

A bar chart of addresses in the block against prefix length from 24 to 32, with two series: total addresses and usable by a host after the two reserved. At a slash 24 that is 256 total and 254 usable. At a slash 30, 4 total and 2 usable. At a slash 31, 2 total and 0 usable. Notes read that a slash 30 gives you two usable hosts and a slash 31 gives you none.
ChartFigure 4. Every subnet loses two addresses, and the small ones lose almost everything

The two reserved addresses are a fixed cost regardless of how large the block is, which means they are invisible on a /16 and fatal on a /31.

RFC 3021 resolves the /31 case by declaring that a point-to-point link has no broadcast address at all, so both of its addresses become usable.

That is why you will see a /31 on router-to-router links and never on a network with hosts on it.

How a Machine Joins a Network

A sequence diagram with three lifelines: a new machine with no address yet, everyone on the whole wire, and a DHCP server. The machine broadcasts DISCOVER asking whether anyone is offering addresses, noted as broadcast because it has no address and knows nobody. The server offers 192.168.0.124. The machine broadcasts REQUEST accepting that one, noted as still broadcast. The server acknowledges, granting it for two hours.
SequenceFigure 5. You cannot address a message to somebody when you have no address yourself

Discover, Offer, Request, Acknowledge. The first and third messages are broadcast, and the reason for the first is obvious.

The reason for the third is not, and it is the more interesting one.

A sequence diagram with four lifelines: a new machine, everyone, server A and server B. The machine broadcasts DISCOVER. Server A offers .124 and server B offers .200, noted as both having now reserved an address for this machine. The machine broadcasts REQUEST for .124 from server A, noted as broadcast so B hears it too. Server B then releases .200 on its own lifeline.
SequenceFigure 6. The request is shouted so the servers you turned down can hear it

Two DHCP servers on one network is a normal, deliberate configuration, and every server that answers has set aside an address it can no longer give to anybody else.

If the request went only to the chosen server, the other reservations would sit there until they timed out, leaking the pool a little at a time.

What Is Actually Shouting on a Quiet Network

A table of six protocols and what each one says. ARP asks who has this IP address, constantly on every network. DHCP asks whether anyone is handing out addresses, when a machine joins and then unicast. mDNS asks whether there is a printer called office, used by Apple's Bonjour and most smart devices. SSDP asks what media players are here, the discovery half of UPnP. NetBIOS asks where the server named FILESVR is, legacy Windows still seen in the wild. Wake-on-LAN says wake up machine with this MAC, the magic packet sent to the broadcast address.
FigureFigure 8. A quiet network is not quiet, and most of the noise is asking for directions

Every one of these exists for the same reason: a device knows what it wants and not where it is.

None of them can be unicast, because the entire point is not knowing who to ask.

Notice the shape. Broadcast is the answer to exactly one question, and once a device has an answer it never needs to ask again until something changes.

The last hop of every unicast packet starts with a broadcast

The previous lesson covered ARP: before your machine can send anything to anybody on the local wire, it has to ask the whole network who owns that IP address. That question is a broadcast.

So the very first step of a unicast conversation is a broadcast, which is worth remembering when you read that broadcast is a legacy concept you will never touch.

Anyone can answer

A sequence diagram with three lifelines: you, everyone, and the attacker. You broadcast asking who has 192.168.0.1, the gateway. The attacker replies that it does and gives its MAC, noted as it is not and nothing in ARP can tell. All of your traffic is then framed to the attacker, who reads it and forwards it on so that nothing appears broken.

Where It Stops, and Why That Matters

A hand-drawn diagram. On the left, subnet A 192.168.0.0/24, where a machine broadcasts to 192.168.0.255 and every host there receives it. In the middle, a router that drops it. On the right, subnet B 10.0.5.0/24, which hears nothing at all and never will. A handwritten note reads that a broadcast domain is defined by where the routers are, not by the cabling.
SketchFigure 10. A router is where broadcast ends, and that is the point of a router

A switch forwards a broadcast frame out of every port it has. A router receives it, sees that it is addressed to this subnet's broadcast address, and stops.

So the reach of a broadcast is exactly one subnet, regardless of how the cables happen to run.

That boundary makes the size of a subnet a real design decision rather than an addressing convenience. Choosing a /16 for a flat office network puts 65,000 possible hosts in one broadcast domain, and every one of them would hear every ARP request from every other.

And you can move the boundary without touching a cable

An isometric diagram of nine machine blocks in three groups of three on one long base slab, coloured green, blue and lavender and labelled VLAN 10, VLAN 20 and VLAN 30. Text reads: one physical switch, three broadcast domains that cannot hear each other, and a router is still required to move between them.
IsometricFigure 11. A VLAN is a broadcast domain drawn in software

Each port is assigned to a VLAN and the switch will not forward between them. A broadcast in VLAN 10 reaches only the ports in VLAN 10.

Two Failures That Are Not the Same Thing

This is the part most treatments get wrong, and the two failures have different causes, different symptoms and different fixes.

The gentle one: overhead

A line chart of relative broadcast overhead against hosts in one broadcast domain, rising in a straight line from zero to a hundred across four thousand hosts. Notes mark that a slash 24 lives at the low end and a flat slash 20 lives near the top.
ChartFigure 13. Overhead grows with the size of the domain, and it never spikes

Every host asks its own questions, so the traffic grows with the number of hosts. Every host also receives everybody else's, so the cost per host grows too.

The curve is smooth. There is no cliff and no moment where something breaks.

That is a real cost on a large flat network and it degrades gently. The fix is not a faster switch, it is a smaller domain.

The catastrophic one: a storm

A broadcast storm is not "a lot of broadcast traffic". It is a specific failure with a specific cause, and calling ordinary overhead a storm sends you looking for the wrong thing.

A hand-drawn diagram of two switches with two cables between them. One broadcast frame travels from switch 1 to switch 2 and back again. A red box reads that an Ethernet frame has no hop limit, so it circulates forever and duplicates. A handwritten note reads that the link saturates in seconds and the network stops, all of it.
Sketch

What Replaced It

IPv6 removed broadcast on purpose

A two-column comparison. On the IPv4 side: ARP broadcasts to everybody, DHCP discovery broadcasts to everybody, all hosts is the dot 255 address, and there is no all routers equivalent. On the IPv6 side, which has no broadcast at all: Neighbour Discovery multicasts to a group derived from the address, DHCPv6 and router advertisements multicast to ff02 colon colon 1 colon 2, ff02 colon colon 1 is a group every host joins, and ff02 colon colon 2 is a group only routers join.
FigureFigure 17. IPv6 has no broadcast address, and everything that used it now uses multicast

The problem with broadcast was never its reach. It was that uninterested hosts still pay, and every one of them is interrupted.

A multicast group is joined voluntarily, so a machine only hears what it asked for. Neighbour Discovery derives its group from the target address, so almost nobody is disturbed by a lookup that does not concern them.

That is the clearest statement of what broadcast is for and what is wrong with it. If you are designing anything one-to-many today, the question is not how to reach everybody. It is how to reach only the ones who want it.

And IPv4 quietly did the same thing

A table of five products with logos and the discovery protocol each uses. Apple's AirPlay, AirPrint and Handoff use mDNS, asking for a name ending in .local. Chromecast uses mDNS, the same mechanism under a different vendor's name. Sonos uses SSDP, the discovery half of UPnP. The Hue bridge uses SSDP, which is how the app finds it unconfigured. Plex uses both, so clients find servers with nothing typed in. A note explains that these strictly use link-local multicast rather than broadcast.
Architecture

When Broadcast Is the Answer

A decision flowchart. Do you know the address of the thing you want? If yes, unicast and nothing here applies. If no, is it on the same local network? If no, you need a directory such as DNS, a registry or configuration, because broadcast cannot reach it. If yes, will most hosts here be uninterested? If yes, multicast to a group, which is what mDNS and IPv6 actually do. If no, broadcast: ARP, DHCP, Wake-on-LAN.
FlowchartFigure 21. Broadcast answers exactly one question, and it answers it locally

The question broadcast answers is: how do I find something whose address I do not know, on this wire?

Only the bottom-right box is broadcast, and everything in it is a bootstrap protocol from the 1980s that still runs today.

That is not a coincidence. Broadcast solves the chicken-and-egg problem, and once you are past it you never need it again.

Broadcast is right when: the address is genuinely unknown, the target is on this network, and either everybody cares or everybody must be given the chance to answer.

Broadcast is wrong when: the target is on another network at any distance, most hosts are uninterested and a multicast group would do, or you already know the address and are broadcasting out of habit.

And a warning about the words. If somebody says the network is having a broadcast storm, ask whether they mean a switching loop or ordinary overhead on a large flat network. Those are different problems. One is fixed by spanning tree or by finding the extra cable, and the other is fixed by making the subnets smaller.

Quick Check

Knowledge Check

6 questions - Score 80% to pass

Q1

A new laptop joins your office wifi with no IP address. How does it find the DHCP server?

Q2

In DHCP, the client already knows which server it wants by the time it sends REQUEST. Why is that message still a broadcast?

Q3

Your network is a flat /16 with about 3,000 machines and it feels sluggish. A colleague says you are having a broadcast storm. Are they right?

Q4

Why does a /31 subnet normally give you zero usable host addresses?

Q5

Your phone cannot find the Chromecast, even though both are on the office wifi. What should you check first?

Q6

Why did IPv6 remove broadcast entirely?

This is worth generalising. When you see a protocol broadcast a message to a recipient it clearly already knows, the question to ask is not why it does not use unicast. It is: who else needs to hear this?

The packet that arrives at the end of that exchange is still on your machine, and it can be printed.

A recorded terminal session showing the output of ipconfig getpacket. The dhcp_message_type is ACK, the yiaddr is 192.168.0.124 described as the address you were given, the server_identifier is 192.168.0.1 described as who gave it to you, and the chaddr is a truncated hardware address. The lease_time is 0x1c20 which is 2 hours 0 minutes, and renewal_t1 is 0xd2e which is 56 minutes. A note explains that t1 is about half the lease and that at t1 the client renews by unicast straight to the server above.
RecordedFigure 7. The lease this machine is actually holding

Three things here are absent from every four-arrow DHCP diagram: how long the lease lasts, when the client will try to renew it, and the fact that the renewal is a unicast.

Only the first exchange is a broadcast. Everything after it is ordinary one-to-one traffic, because by then the client knows who to talk to.

Broadcast solved exactly one problem here, and it was bootstrapping.

SequenceFigure 9. A broadcast question can be answered by anybody who feels like it

ARP has no authentication, no signatures and no concept of who is entitled to reply. You ask the network a question and accept the first answer that arrives.

Forwarding the traffic on afterwards means nothing looks broken from either end.

This is the strongest practical argument for treating the local network as untrusted, which is most of what people mean by zero trust. It is also why encryption in transit matters on a network you own outright: being on the same wire as an attacker is enough.

Moving traffic between VLANs requires a router, exactly as between physically separate networks.

This is why the guest network on an office wifi cannot see the printers. It is not a firewall rule and there is nothing to troubleshoot in the app: it is a different broadcast domain, so the discovery packet never arrives.

Shouting into somebody else's network

A hand-drawn diagram. Directed broadcast: a packet from outside addressed to 192.168.1.255 with the victim's address forged as the source. An attacker sends one ping, 250 machines all reply, and the victim drowns. Handwritten notes read that one packet in becomes two hundred and fifty out and none of them from the attacker, and that routers stopped forwarding directed broadcast by default, which killed it.
SketchFigure 12. The attack that made every router stop forwarding broadcast inward

A directed broadcast is a packet addressed to a subnet's broadcast address, sent from outside that subnet. The attacker forges the source so every reply goes to the victim instead.

The victim sees traffic from hundreds of innocent machines and cannot filter it by source, because none of the sources is lying.

The fix was a default change rather than a protocol change: routers now discard directed broadcasts unless somebody explicitly configures otherwise. Worth knowing because the same amplification pattern reappears in and NTP reflection attacks, with an identical shape and a different protocol.

Figure 14. A storm is a loop, not a busy network

IP packets carry a hop limit. Ethernet frames carry nothing of the sort, so a frame that reaches a switch it has already passed is simply forwarded again.

Every pass multiplies the copies, because each switch floods it out of every port. The result is not slow, it is total, and it happens faster than anyone can react.

Two switches and one accidental extra cable is enough. That is why an entire protocol exists to prevent it.

A graph of three switches. Switch 1 is the elected root and connects to switch 2 and switch 3, each of which connects to hosts. The link between switch 2 and switch 3 is drawn as a dotted line labelled this link is BLOCKED on purpose.
FlowchartFigure 15. Spanning tree finds every loop and turns one link in each of them off

The switches elect a root, compute a loop-free tree, and block whatever links are left over. The blocked link is physically connected and carrying nothing at all.

It is not wasted: if an active link fails, the tree is recomputed and the blocked one comes up.

Two consequences show up in real work. Recovery on the classic version takes seconds rather than milliseconds, which is why faster variants exist. And you cannot get more by adding a second cable between two switches, because one of them will be blocked. Link aggregation is the tool for that instead.

And one thing that looks like broadcast and is not

A sequence diagram with three lifelines: machine A, the switch which learns as it goes, and machine B. A sends a unicast frame for B's MAC, noted as the switch never having seen B send anything. The switch sends it out of every port, noted as flooding because it does not know which port B is on. B replies, so now the switch knows. The next frame for B goes to one port only.
SequenceFigure 16. A switch sending a frame everywhere is not always a broadcast

A switch builds its table by noticing which port each source address arrives on. Until it has heard a machine speak, it does not know where that machine is.

A frame for an unknown destination is therefore sent out of every port, which looks exactly like a broadcast on the wire.

Worth separating clearly in your head. A broadcast is addressed to everybody. Flooding is addressed to one machine and delivered to everybody because the switch is ignorant. A machine that stays silent long enough falls out of the table and starts getting flooded again.

Figure 18. Zero configuration is one device shouting and another one answering

Nobody types an address for a printer or a speaker any more, and this is what replaced it. mDNS is with no server: a device asks the local network and the owner answers.

All of it is local only, because the mechanism cannot cross a router by design.

That last line is a diagnostic. When a device is "not found", check whether the two ends are in the same broadcast domain before checking anything else. On a guest network, a separate VLAN or a VPN, the discovery packet never arrives, and no amount of restarting the app will change that.

The case with no alternative at all

A figure explaining Wake-on-LAN. The machine you want is powered off, has no IP address, is in nobody's ARP table and cannot be addressed, but its network card is still drawing power and listening. The magic packet is six bytes of FF followed by the target MAC repeated sixteen times. It is sent to the subnet's broadcast address because there is no other way to reach a machine with no address. There is no acknowledgement and no reply.
FigureFigure 19. How do you send a message to a computer that is switched off?

A powered-off machine has no address and cannot appear in any table, so it cannot be addressed by any normal means. Its network card stays powered and inspects every frame on the wire.

The magic packet is deliberately unmistakable, and nothing acknowledges it, because the thing you are talking to is not running yet.

This is also why Wake-on-LAN does not work from the internet without a helper inside the network. The packet has to be broadcast on the target's own subnet, and no router will carry a broadcast in from outside, for exactly the reason the smurf figure gave.

How the industry stopped worrying about it

Two rows. The old way: one large flat layer-2 network per row of racks, meaning one broadcast domain with thousands of hosts and spanning tree everywhere. The modern way: routing all the way down to each rack, so a broadcast domain is one rack and it is small. Below, three vendor logos: Cisco, NVIDIA slash Mellanox and Ubiquiti. A note reads that every rack becomes its own subnet with its own router, so no broadcast reaches more than a few dozen machines and no spanning tree has to block a cable.
ArchitectureFigure 20. The answer turned out to be more routers, not better switches

A large flat layer-2 network was the old design, and it made broadcast a real cost while also requiring spanning tree, which meant paying for cables that carried nothing.

Routing to each rack makes every broadcast domain a few dozen machines, and lets every cable carry traffic.

That is worth carrying well beyond networking. The problem was not that broadcast was too expensive. It was that the domain was too big, and the fix was to change the boundary rather than to optimise the thing inside it.