The Morning One Region Went Quiet
Let me start with a real morning.
On Tuesday, 7 December 2021, at 7:30 AM Pacific time, an automatic job inside Amazon Web Services started adding more computers behind one of its own programs. (AWS calls each of its programs a service.) That is a normal thing to do. This time it set off a flood of new connections inside a private network, the cables and boxes that carry messages between AWS's own computers. The boxes in the middle, which pass messages from that network to the main one, filled up. Requests that would normally take a moment started to time out, and the programs that timed out tried again, which made the flood bigger.
This happened in one place: the AWS region called us-east-1, in Northern Virginia. (A region is a group of data centres in one area; more on that soon.) Here is how AWS described the start, in the summary it published afterwards.
This is the real page, not a drawing of it. Notice the last two sentences: delays caused retries, and retries caused more delay. A small problem fed itself until it filled the network.
Here is what people saw, from the same summary. People could not log in to the AWS console, the website customers use to manage their things, for that region. The tools for starting new servers, the computers that run programs, returned errors, and starting a new server did not work again until 2:40 PM. The tools for changing records did not work from 7:30 AM to 2:30 PM. DNS is the internet's address book: it turns a name like example.com into the address of a computer. Amazon Connect, the AWS call-centre service, failed more calls and chats than normal until 4:41 PM. A service that passes messages between programs ran late until 6:40 PM. Even the AWS status page, the page that tells customers what is wrong, could not move over to its spare copy in another region as it was designed to.
Some things kept working. Servers that were already running kept running. That detail matters later.
The network was healthy again at 2:22 PM. That is almost seven hours. Some services took until the evening, about eleven hours after the start.
The data in Virginia was not lost that day. What was lost was the ability to change things there: start servers, edit DNS, log in. Running servers carried on, which is why a plan that needs the broken region's controls is a weak plan.
So here is the question this lesson answers. If every copy of your data lives in one region, and that region has a bad day, what can you do? The honest answer is: wait. The only way to have a second choice is to keep a copy of your data somewhere far enough away that the same bad day does not reach it. Keeping that copy up to date is the subject of this lesson.
In one sentence: cross-region keeps a live copy of your data in a second cloud region or data centre, so one region's bad day does not stop you. The copy is updated either synchronously, where every save waits for the far copy, or asynchronously, where the save is confirmed first and copied a moment later.