We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In January 2017, a senior engineer at GitLab accidentally deleted 300 GB of production data from a PostgreSQL primary. The team turned to their backups. They had five separate backup mechanisms configured: regular pg_dump, filesystem-level snapshots, LVM snapshots, Azure disk snapshots, and S3 uploads.
All five were broken. Some had been silently failing for months. Others produced files that could not actually be restored. The only method that held usable data was a pg_dump that happened to have run 6 hours earlier, and even that was discovered by luck rather than by any verification system.
Backup verification exists to prevent this exact scenario. The goal is simple: prove, automatically and on a schedule, that your backups can actually produce a working database. A backup that has never been restored is a hope, not a guarantee. Verification converts hope into a measured, auditable fact.
Verification is a pipeline that runs after every backup (or on a fixed schedule) and answers one question: can this backup produce a working database?
| Verification Layer | What It Catches | Cost |
|---|---|---|
| Checksum comparison | Bit rot, truncated uploads, tampered objects | Seconds of CPU, zero extra I/O if computed during dump |
| Restore to ephemeral DB | Corrupt dump files, missing extensions, broken format | $0.40 per run (25 min on r6i.4xlarge for 500 GB) |
| Sanity queries on restored data | Missing rows, schema drift, truncated tables | Seconds of query time after restore |
| Application boot test | Broken collations, missing migrations, incompatible versions | Minutes of app startup against the clone |
Each layer catches problems the previous layer misses. A file with a perfect SHA-256 hash can still contain a corrupt pg_dump that fails on restore. A clean restore can still be missing half the rows if the backup captured a replica that was behind on . Only the full pipeline, from hash through application boot, gives real confidence.
The output of verification is a metric: pass or fail, with restore duration. Treat it as an SLO (for example, 99% of nightly backups successfully verify within 24 hours). Alert when the rolling rate drops below the target.