We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Data Governance Compliance
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Data Governance Compliance
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Your system receives customer data from three sources: a CSV upload from the sales team, a JSON webhook from Stripe, and a SOAP XML response from a legacy ERP system. Each source has different field names, date formats, and data structures.
"John Smith", "john@acme.com", "04/03/2026"{ "customer": { "name": "John Smith", "email": "john@acme.com", "created": 1775280000 } }<Customer><FullName>SMITH, JOHN</FullName><Email>JOHN@ACME.COM</Email><Date>2026-04-03T00:00:00Z</Date></Customer>Same person. Three completely different representations. Your application needs one consistent format. Data transformation converts all three into your canonical model: { name: "John Smith", email: "john@acme.com", createdAt: "2026-04-03T00:00:00Z" }.
Netflix processes over 1.5 trillion events per day. Each event passes through transformation pipelines that normalize timestamps from different timezone formats, map device-specific field names to a unified schema, and enrich raw events with metadata from lookup tables. Without transformation, their analytics would be meaningless, comparing data in different formats is like comparing distances measured in miles and kilometers without conversion.
| Transformation | What It Does | Example |
|---|---|---|
| Mapping | Rename fields from source to target | customer.fullName to user.name |
| Type conversion | Change data types | Unix timestamp 1775280000 to ISO date string |
| Format standardization | Unify different formats | "04/03/2026", "2026-04-03", "April 3, 2026" all become ISO 8601 |
| Normalization | Reduce redundancy, enforce consistency | "SMITH, JOHN" to "John Smith" (proper case) |
| Enrichment | Add derived data | Calculate ageGroup from dateOfBirth |
| Structural | Change the shape of data | Flatten nested JSON, pivot rows to columns |
| Filtering | Remove irrelevant fields | Strip internal IDs before sending to client |
| Aggregation | Combine multiple records | Roll up daily sales into monthly totals |
Field mapping is the most common transformation. Every API integration involves it. Stripe calls it customer.name. Salesforce calls it Account.Name. HubSpot calls it contact.firstname. Your database calls it users.full_name. The mapping layer translates between all of them.