We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Data Governance Compliance
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Data Governance Compliance
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
You receive a customer export from a legacy system. Here is what the "phone" column looks like:
(555) 123-4567
555.123.4567
+1-555-123-4567
5551234567
555 123 4567
N/A
NONE
-
555-1234
Same field, eight different formats, two non-phone values, and one incomplete number. This is not an edge case. This is what real data looks like. Every system that has been in production for more than a year has data like this.
Data cleansing is the process of detecting and correcting (or removing) corrupt, inaccurate, and inconsistent records. It is tedious, unglamorous work, and it is absolutely necessary. A machine learning model trained on dirty data makes dirty predictions. A report built on dirty data produces wrong numbers. An analytics pipeline processing dirty data wastes money on garbage.
Airbnb's data team found that 30% of engineering time on data projects went to cleansing. Not building models, not creating dashboards, just getting the data into a state where it could be used. They built an internal platform called Minerva specifically to standardize cleansing across teams.
| Operation | What It Does | Example |
|---|---|---|
| Standardization | Convert to a consistent format | Phone numbers to E.164 format: +15551234567 |
| Deduplication | Remove or merge duplicate records | "John Smith" and "Jon Smith" at same address: merge |
| Missing value handling | Fill, flag, or remove nulls | Empty "country": infer from phone prefix or address |
| Outlier treatment | Handle values outside expected ranges | Age = 999: flag for review or set to null |
| Type correction | Fix data type mismatches | "42" (string) to 42 (integer) |
| Trimming | Remove leading/trailing whitespace | " John Smith " to "John Smith" |
| Case normalization | Standardize casing | "john@EMAIL.com" to "john@email.com" |
| Spell correction | Fix common misspellings | "Califronia" to "California" |
| Referential repair | Fix broken foreign key references | order.customer_id = 999 (does not exist): flag for review |