Continuous Backup

Nightly backups have a fatal flaw: you lose everything since last night. If a destructive query runs at 4 PM and your last backup was at 2 AM, fourteen hours of transactions are gone. For a payments database processing $2 million per hour, that is $28 million of financial records you cannot reconstruct.

Continuous backup eliminates this gap by capturing every single write as it happens. The mechanism is simple in principle: the database already writes every change to a transaction log (WAL in Postgres, binlog in MySQL, redo log in Oracle) before acknowledging the commit. Continuous backup ships that log to remote storage in near-real-time, giving you the ability to restore to any second in the database's history.

GitLab learned this lesson publicly on January 31, 2017. An SRE accidentally deleted 300 GB of production data. Their nightly pg_dump was 6 hours stale, their WAL archive had silently stopped working, and the postmortem became one of the most-read incident reports in our industry. If the WAL archive had been healthy, recovery to 60 seconds before the deletion would have been straightforward.

The technology behind continuous backup is not new. Postgres has supported WAL archiving since version 8.0 in 2005. What changed is the tooling around it: wal-g (born at Citus Data, now used by GitLab, Zalando, and Adjust) automates the compression, encryption, and upload pipeline so teams do not have to build it from scratch.

Continuous backup is not a replacement for full backups. It is a complement. You need both: periodic full backups as a baseline and a continuous WAL stream to fill the gaps between them.

The continuous backup pipeline has three components:

1. Base backup taken weekly with pg_basebackup or wal-g backup. This is the starting point for any restore.
2. WAL archive stream shipping sealed 16 MB segments to S3 via archive_command or wal-g. This is the continuous part.
3. Lag monitor checking that the newest segment in S3 is less than 60 seconds old. This is the safety net that catches silent failures.

Without all three, continuous backup is incomplete. A WAL stream without base backups means restoring requires replaying months of logs. A base backup without WAL means you lose everything since the last base. A pipeline without monitoring means you discover failures during the incident, not before.