We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
5 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Open LinkedIn and click on any job posting. Within a second you see how many people viewed it today, how many applied in the last hour, and where the recruiter sits in the funnel. None of that is coming from a nightly job. It is coming from Apache Pinot, and the query ran in 70 milliseconds against 200 million rows.
Real time analytics is the discipline of answering aggregate questions about data that is still moving. Not reports that run at 3 AM and land in a warehouse. Not dashboards that refresh every five minutes. Sub second queries on data that arrived a few seconds ago. LinkedIn built Pinot for exactly this use case in 2013 after Druid (originally from Metamarkets) proved the model worked for ad analytics at scale. ClickHouse came out of Yandex Metrica the same year. Rockset and Materialize added on top of the idea more recently.
The reason the category exists at all is that regular OLTP databases cannot scan 200 million rows in 70 milliseconds and warehouses like Snowflake cannot answer a query in under a second no matter how small the data is. Real time analytics engines sit in a new spot on the versus freshness curve: data that is at most a few seconds old, queries that return in under 100 milliseconds, and a SQL interface people already know.
By the end of this lesson you will understand how Pinot ingests from into columnar segments, how a broker fans a query across servers and why p99 stays under 100 milliseconds, and why these engines exist separately from both OLTP and warehouses.
The first question every new Pinot user asks is how data that was in a second ago becomes queryable. The answer is a consuming segment pattern Druid pioneered and Pinot refined.
LinkedIn runs Pinot behind Who Viewed Your Profile, Feed Analytics, and dozens of member facing dashboards. Their production numbers tell the story: around 150 billion events a day, p99 query around 70 milliseconds, segment freshness in the single digit seconds. The trick is that a Pinot realtime server does not wait for data to land in storage before making it queryable. It indexes in memory while consuming.
Each Kafka partition is assigned to exactly one realtime server. That server opens a consumer, builds indexes on the fly as rows arrive, and holds the whole thing in a consuming segment in JVM heap. Queries that hit this segment scan the in memory indexes directly. When the segment crosses a row count or size threshold, Pinot flushes it to a columnar file on S3 or HDFS and the segment becomes committed and immutable. A new consuming segment starts and the cycle continues.
Watch a click event travel from Kafka into a consuming segment, get indexed on arrival, flush to deep storage when full, and become readable through the broker.
Pick the flush threshold carefully. LinkedIn caps consuming segments around 500 MB so that a flush does not stall queries and GC pauses stay under 200 milliseconds. A threshold that is too high means long JVM pauses during flush. A threshold that is too low means too many tiny files on S3 and the broker spends more time planning than scanning. Start at 500 MB or 5 million rows and tune from there.
placement matters for availability. Pinot places committed segments on two additional servers so a single node failure does not take down a query path. But a consuming segment only exists on one server by design because the Kafka partition is assigned to one consumer. If that server crashes, the controller reassigns the partition and a new consuming segment rebuilds from the committed Kafka offset. For the few seconds of replay, queries see slightly stale data for that partition range. Netflix handles this by over provisioning and running with enough headroom that a single server replay is unnoticeable.
Do not put anything in Pinot that does not need to be fresh. It is more expensive per row than a warehouse because of all the indexing work. Keep it for dashboards a human is staring at or alerts that fire on live data. Historical analysis that tolerates hour old data belongs in BigQuery or Snowflake.
The most common mistake is treating Pinot like a general purpose database. It has no joins worth speaking of, no transactions, and no UPDATE semantics. The data model is append only. If you need to correct a row, you insert a new one with a tombstone column and filter the old one out at query time. Every team that tries to add UPDATE support discovers the columnar segment format fights them.