Name: System Design Masterclass
Availability: InStock

Every rule so far in this chapter ends with one answer. A path matches one backend. A weight picks one. A failover uses one until something says otherwise.

Multi-value routing ends with several, and lets the client decide.

Four rows. Path, host and header: your proxy decides, giving one answer and it is the right one. Weighted: your proxy decides, one answer chosen by a coin. Failover: your health check decides, one answer until a signal changes it. Multi-value: the client decides, several answers and it picks. A closing line reads: the behaviour of this one is not defined by your configuration, it is defined by whatever library the client happens to use. — FigureFigure 1. Every other rule in this chapter ends with one answer. This one ends with a list
Handing over several options moves the load spreading, the retrying and the failure handling out of your infrastructure and into somebody else's client.

That is the trade. You get client-side resilience for free, and you give up any ability to say what happens, because the algorithm lives in code you did not write and cannot deploy.

So the useful question is what clients actually do.

A recorded terminal session. A server is listening on 127.0.0.1 and curl is given two addresses in three orders. With 127.0.0.1 first, the request reaches 127.0.0.1 in 1.6 milliseconds. With an address that refuses first, it reaches 127.0.0.1 in 0.8 milliseconds. With 192.0.2.1, which goes nowhere, first, it reaches 127.0.0.1 in 5,009.8 milliseconds. The system resolver then returns the same address order five times in a row. — RecordedFigure 2. Three orderings of the same list, and every one returned 200
Client-side failover works. What varies by three orders of magnitude is the price.

A bar chart with three bars. Working address first, 1.6 milliseconds. Refused first, 0.8 milliseconds. Blackholed first, 5,009.8 milliseconds. A note reads: all three succeeded. — ChartFigure 3. The same request took 1.6 milliseconds or five seconds
The client did the right thing in all three cases. The wait is its connect timeout.

A hand-drawn comparison. Connection refused: the machine is up and nothing is listening, and the client knows at once. Silence: the machine is gone or a firewall is dropping, and the client waits, and waits. Two notes read: measured, 0.8 milliseconds against 5,009.8, and which one you get is decided by how the machine died. — SketchFigure 4. A refusal is a fast answer and silence is not an answer at all
Client-side failover is quick when the dead address says so, and as slow as the connect timeout when it says nothing.

A decision flowchart. When a backend of yours dies, what does a client see? Connection refused: fast, the client moves on in about a millisecond, this is the good one. Nothing at all: slow, the client pays its whole connect timeout before trying anything else, which happens when the host is gone or a firewall drops instead of rejecting. It answers badly: worst, no connection error at all, so nothing in the client will try elsewhere. — FlowchartFigure 5. How your backend dies decides how long every client waits
The client's behaviour is fixed. What varies is the signal your infrastructure gives it, and some of that is a choice you make in a firewall rule.

The firewall row is actionable today. A rule that rejects gives every client a fast, correct signal; a rule that drops makes every client wait. Dropping is the right default facing the internet and the wrong one facing your own clients.

A line chart. The x axis is dead addresses tried before a working one, zero to three. The y axis is seconds of waiting before the request starts, zero to ninety. A 30 second connect timeout rises to 30, 60 and 90 seconds. A 2 second timeout rises to 2, 4 and 6. A note reads: multiplication, not a measurement. — ChartFigure 6. Every dead address costs one connect timeout
This chart is arithmetic, not a measurement. The measured point is in the capture above: one silent address cost 5,009.8 milliseconds.

The default connect timeout in most libraries assumes an unreliable public network and a patient user. Inside a data centre, where a healthy connect takes under a millisecond, a two second timeout is already a thousand times the expected value and is generous.

A hand-drawn sequence. Start connecting to the first address. Wait about 250 ms, not for a timeout. Start the next one in parallel. Below: whichever connects first wins, and the other attempt is dropped. A note reads: a browser hides a dead address, a server-side client usually does not. — SketchFigure 7. Browsers race the addresses instead of waiting
The algorithm has a name and a specification, and it is why a dead address is invisible on the web and expensive everywhere else.

Check whether your own clients do this before relying on multi-value routing for failover. Browsers do. Many server-side libraries try one address, wait for the whole connect timeout, and only then try the next.

Multi-Value Routing

The rest of this lesson

Continue this lesson

The Rule That Ends With a List

What the Client Actually Does