We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
failover is the most widely deployed failover mechanism on the internet because it requires zero changes to your application code. Configure health checks on your DNS provider, point your domain at two IPs, and the provider stops returning the unhealthy IP when the health check fails. Route 53, Cloudflare, NS1, and every major DNS provider offer this out of the box.
The appeal is simplicity. The weakness is speed. DNS failover is fundamentally limited by the of your DNS records and the caching behavior of resolvers between your DNS provider and your end users. A 60 second TTL means clients that just resolved your domain will keep hitting the dead IP for up to 60 seconds. Mobile carriers and corporate proxies often cache DNS responses far beyond the TTL you set, stretching the recovery tail to 5 or even 15 minutes.
Cloudflare published research in 2020 showing that 5 percent of global resolver traffic honored TTLs below 60 seconds for an additional 10 minutes beyond the intended expiry. Netflix and Stripe both publicly describe DNS failover as the outer safety net, not the primary failover path. Their inner layer uses anycast BGP withdrawal or global load balancers that recover in single-digit seconds. DNS catches the long tail that the inner layer misses.
Understanding DNS failover matters because it is the foundation that every higher-level failover mechanism builds on. Even active-active architectures with global load balancers rely on DNS at the outermost layer.
| Phase | What Happens | Typical Timing |
|---|---|---|
| Health check probing | provider pings primary endpoint every 10 to 30 seconds | Continuous |
| Failure detection | N consecutive failed checks triggers an unhealthy verdict | 30 to 90 seconds (3 failures at 10 to 30 second intervals) |
| DNS record update | Provider stops returning the primary IP, starts returning the secondary | Near instant on the provider side |
| Resolver cache expiry | Cached DNS responses at recursive resolvers expire based on | 10 to 300 seconds depending on configured TTL |
| Client reconnection | Clients with cached connections to the dead IP time out and reconnect via fresh DNS lookup | 30 seconds to several minutes depending on keep-alive settings |
| Long tail drainage | Mobile carriers, corporate proxies, and stubborn resolvers that ignore TTL gradually expire | 5 to 15 minutes |
Total recovery time from failure to 100% of traffic on the secondary: 2 to 5 minutes for well-configured setups, 10 to 20 minutes in the long tail. This is why DNS failover is appropriate for services with an RTO of 5 minutes or more but insufficient for services that need sub-minute recovery.