We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Spark Structured Streaming does not actually stream. It runs a tight loop of small batches and pretends the result is a stream. When the model first shipped in 2016 the streaming purists laughed. A decade later Databricks is the default streaming platform at a large fraction of enterprises, which tells you the trade off is better than the purists thought.
Micro batching takes the whole Spark machinery (DataFrames, Catalyst optimizer, , Delta Lake) and applies it to small windows of streaming data. The driver wakes up every trigger interval (default 500 ms), reads the new Kafka offset range, plans a Spark job for just that slice, and ships the plan to long lived executors. Each trigger is a full Spark job with a plan, a shuffle, and a commit. Cheap because the executors are warm, the JIT is hot, and the plans are cached.
The reason micro batching wins more often than it deserves is that most analytics pipelines do not actually need sub second . They need exactly once semantics, good SQL, and reliable writes to a warehouse or lake. Spark provides all three with one framework that is already deployed for batch. Flink and Kafka Streams are better at low latency and stateful joins, but they are a second system that also has to be run, monitored, and staffed.
By the end of this lesson you will understand exactly how Spark fakes streaming, what the trade off versus true streaming actually costs in latency, and when each answer is right.
The mental model for Spark Structured Streaming is a driver that sits in a loop. Every trigger interval it wakes up, asks for the current end offsets, subtracts the last committed offsets to get a range, and calls that range the batch. It plans a Spark job, ships it to the existing executors, waits for them to finish, writes the result to the sink, and commits the new offsets to the checkpoint directory. Then it sleeps until the next trigger.
Databricks publishes recommendations that are worth remembering. For ETL into Delta Lake, use a 30 second trigger. For dashboard feeds, use 1 to 5 seconds. For anything under 1 second, you are fighting the system because each batch has a minimum overhead of a few hundred milliseconds for planning and scheduling, regardless of how little data is in the batch. The continuous mode that tried to remove the trigger loop has been experimental for years and Databricks does not recommend it in production.
The huge advantage of micro batching is that the full Spark toolbox is available. Joins to static Delta tables, window functions, complex , MERGE INTO a Delta table with exactly once semantics via the transaction log. Anything you can do in batch Spark works in streaming Spark with almost no code change. That is the reason Databricks customers default to it even when their latency budget could afford true streaming.
Watch the Spark driver wake up every 500 ms, read from Kafka, plan a job, ship work to long lived executors, write to Delta, and commit the checkpoint.
Tune the trigger interval to match your actual budget rather than the smallest value that works. Every trigger has overhead: query planning, task scheduling, committing the checkpoint. At 500 ms trigger you might spend 30 percent of CPU time on overhead. At 10 seconds you spend 2 percent. Databricks recommends the longest trigger that meets the SLA. For most ETL pipelines that is 30 seconds or longer.
Use Delta Lake as the sink when you can. The Delta transaction log gives you exactly once semantics for writes and a physical layout the next batch Spark job can read efficiently. Raw Parquet writes to S3 without Delta lose the exactly once guarantee on retries because Spark cannot distinguish a successful write from a partial one. Every team that tried writing plain Parquet to S3 directly learned this in a data quality incident.
Size the executors for the peak batch, not the average. A burst that triples the event rate triples the per batch data volume, which means you either keep peak sized executors running all the time or accept that bursts will cause batch overruns (a batch taking longer than the trigger interval, which causes queueing and growing latency). Databricks autoscaling helps but cannot react within a single trigger interval.
The common mistake is treating the trigger interval as a latency guarantee. It is not. End to end latency includes the time from event production to Kafka, Kafka to Spark read, the trigger wait, batch planning, execution, and sink write. A 500 ms trigger with a 300 ms batch execution means a median end to end latency around 700 ms and a p99 closer to 1.5 seconds. If your SLA is 100 ms, Spark is not the answer.