A Weight of One Hundred and Zero
Failover is the simplest rule in this chapter. Use the primary. If something says the primary is broken, use the standby.
The swap is instant and trivial. Everything difficult about this lesson is in deciding that the moment has come, and in what the standby has been doing meanwhile.
Two consequences follow immediately. The detection is the whole mechanism, and the standby carries no traffic, which means nobody has tested it.
The Window Between the Failure and the Detection
The failover itself takes no time at all. The window before it is the outage, and its length is a number you chose.
Here is that window, measured on a real with real timing.
Eleven of forty requests failed on the way to a correct, fast failover. That is not a bug. That is the window.
The second block is the more important one, and it is about what the check is actually asking.
A process can answer a health endpoint perfectly while being completely unable to serve a request. The capture shows a shallow check never noticing that at all.
There is a real tension here. A deep check sees the failure that matters, and it can also fail for a shared reason, marking every instance unhealthy simultaneously and turning a degradation into a complete outage. The usual balance is a shallow check wired to the and a deep check wired to alerting.
The probe configuration is the part people tune. The downstream of the decision is the part that dominates.
Do this multiplication before choosing a mechanism. If the sum is in minutes and the request rate is high, no amount of probe tuning helps.