We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
2 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Database Types Storage
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Database Types Storage
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Imagine you're running a logging pipeline that ingests 500GB per day. After a month, you've got 15TB of logs. After a year, 180TB. If you keep everything on fast SSDs, your cloud bill starts looking like a mortgage payment.
So you decide to move old data to cheaper storage. Great plan. But now someone has to decide when to move it, where to move it, and how to move it without breaking any running queries. And they have to do it every single day. Forever.
That's the problem tiered storage solves. Instead of manually shuffling data around, you define policies. "move indexes older than 7 days to warm nodes, older than 30 days to cold nodes, delete after 90 days", and the system handles everything automatically.
Apache introduced tiered storage in KIP-405 specifically because brokers were drowning in retention data. Confluent reported that some customers were spending more on Kafka storage than on compute, a completely upside-down cost ratio that tiered storage flipped back.
Tiered storage separates the where data lives decision from the how data is accessed decision. The application doesn't need to know which tier holds a particular piece of data, the storage system abstracts that away.
Here's the general model:
| Component | Role |
|---|---|
| Policy engine | Defines rules for when data moves between tiers (age, access count, size) |
| Data mover | Background process that physically relocates data based on policies |
| Metadata catalog | Tracks where every piece of data currently lives |
| Transparent access layer | Routes queries to the correct tier without the application knowing |
The advantage of this design is that reads still work the same way. A query for a 90-day-old log entry goes through the same API as a query for today's entry, the storage system transparently fetches it from whatever tier it landed on.
The trade-off: varies by tier. A hot-tier read takes milliseconds. A cold-tier read might take seconds. Your application needs to tolerate that variance.