We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Devops Cicd
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Devops Cicd
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A team migrated from MongoDB to PostgreSQL. They wrote a migration script, ran it over a weekend, and declared success on Monday. Two weeks later, customer support started getting complaints. 10,000 customer records were missing. The migration script had silently failed on records with special characters in the name field (em dashes in names, Unicode apostrophes, emoji in profile bios). No validation, no error logging, no reconciliation. The data existed in the old database but never made it to the new one.
Data migration is one of the highest-risk operations in software engineering. You are moving data that may be irreplaceable between systems. Any bug, encoding issue, or edge case can cause data loss. Stripe has written publicly about their PostgreSQL migration taking over a year of dual-running both databases. Shopify migrated their largest merchants to a new system over 18 months. GitHub migrated from MySQL to Vitess over multiple years. None of these were "run a script over the weekend" operations.
Every successful migration follows the same structure, regardless of the source and target systems:
| Phase | Activities | Duration |
|---|---|---|
| Assessment | Map source schema to target. Identify data quality issues, encoding problems, missing fields. Profile the data. | Days to weeks |
| Transformation rules | Define how every field maps, converts, and cleans. Handle nulls, defaults, type conversions. | Days |
| Dry run | Run migration on a copy of production data. Validate completeness. Measure duration. | Hours to days |
| Migration execution | Execute with thorough logging. Track every record: success, failure, skipped. | Hours to weeks |
| Validation | Row counts, checksums, sample verification, business logic checks. | Hours |
| Reconciliation | Compare source and target. Identify and fix discrepancies. | Hours to days |
| Cutover | Switch application to new system. | Minutes |
| Parallel run | Keep old system running as fallback. Monitor both. | Days to weeks |
Validation is non-negotiable. Run every single one of these:
| Check | What It Verifies | Catches |
|---|---|---|
| Row count comparison | Same number of records | Silently dropped records |
| Checksum comparison | Hash of key columns matches | Corrupted or truncated data |
| Random sample verification | 100+ random records manually compared | Transformation bugs |
| Business logic checks | Computed values (totals, aggregates) match | Calculation errors |
| Edge case verification | Records with nulls, special chars, max-length strings, zero values | Encoding and boundary bugs |
| Referential integrity check | All foreign keys resolve in the target | Orphaned records |