We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
In short
Stream processing is a way of handling data continuously as each event arrives, computing results within milliseconds to seconds instead of waiting to collect data and process it in scheduled batches. It powers things like fraud detection, live dashboards, and real-time recommendations by treating data as an endless flow rather than a fixed-size file.
In short
Stream processing is a way of handling data continuously as each event arrives, computing results within milliseconds to seconds instead of waiting to collect data and process it in scheduled batches. It powers things like fraud detection, live dashboards, and real-time recommendations by treating data as an endless flow rather than a fixed-size file.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Stream Processing across a run of lessons, not one page. These 4 alone are about 135 minutes of step-by-step reading, every one with a quiz.
Process data as it arrives, real-time analytics, fraud detection, and event-driven architectures.
Design a real-time analytics system - stream processing, WebSocket-powered dashboards, aggregation pipelines, and windowed computations at scale.
Transform events independently, filtering, mapping, and routing without maintaining state.
Maintaining state across events, aggregations, joins, and pattern detection in streaming data.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Stream Processingpay once,
yours for life
In batch processing you collect data over a window, say an hour or a day, then run a job over the whole pile at once. Stream processing flips that. Each record, click, payment, sensor reading, log line, is handled the moment it shows up. The pipeline never finishes because the data never stops coming.
The unit of work is an event. An event is a small immutable fact: user 4471 clicked at 14:02:13, card ending 8829 was charged 49.00, temperature sensor 12 read 81 degrees. A stream processor reads these events from a log like Apache Kafka or Amazon Kinesis, applies logic to each one, and emits results downstream within milliseconds.
Because there is no end to the stream, you compute over windows. A tumbling window groups events into fixed five-minute buckets. A sliding window recomputes a one-minute total every ten seconds. A session window groups activity until a user goes quiet for thirty minutes. Windows are how you turn an infinite flow into answerable questions like how many logins failed in the last minute.
A stream processor like Apache Flink or Kafka Streams runs as a long-lived job spread across many workers. The input stream is split into partitions, and each worker owns a slice. Events for the same key, the same user or the same card, always route to the same worker so counts and aggregates stay consistent.
State is the hard part. To count failed logins per user over five minutes, the processor must remember running totals between events. Flink keeps this state in an embedded RocksDB store on local disk and periodically writes a consistent checkpoint to durable storage like S3. If a worker crashes, the job restarts from the last checkpoint and replays events from the saved offset in Kafka, so nothing is lost and nothing is double counted. That guarantee is called exactly-once processing.
Time is the other hard part. Events arrive out of order because of network delays and retries. Serious systems use event time, the timestamp baked into the event, not the wall clock when it was received. A watermark tells the processor it is reasonably safe to assume no events older than a certain time will still arrive, so it can close a window and emit the result. Late events that slip past the watermark can be dropped or sent to a side output.
Reach for stream processing when the value of an answer decays fast. Blocking a stolen credit card is worthless an hour later, so fraud scoring has to happen in the few hundred milliseconds before the charge clears. Same logic applies to live dashboards, alerting on error spikes, dynamic pricing, and feeding fresh features to a recommendation model.
The cost is real complexity. A batch job that fails just gets rerun tomorrow. A streaming job runs forever, so you have to design for late data, out-of-order events, state that grows without bound, schema changes on a live pipeline, and the question of what exactly-once even means when results have already been sent to an external system. Debugging a continuous job is harder than rerunning a script over yesterday's file.
If you do not need an answer in seconds, batch is usually cheaper, simpler, and easier to reason about. Many teams run both: a batch pipeline computes the slow, fully accurate numbers overnight, while a stream pipeline serves the fast, approximate numbers in real time. That hybrid is common enough to have a name, the Lambda architecture. The Kappa architecture is the opposite choice: it drops the batch layer and runs everything, including reprocessing, as a stream.
Picture a bank scoring transactions. Every swipe lands as an event on a Kafka topic partitioned by card number. A Flink job reads the topic and, for each card, keeps state about recent spending: the last ten transactions, the running total in the past minute, the usual home location.
When a new charge arrives, the job compares it against that state in a single pass. A 2000 dollar purchase in another country two minutes after a local 4 dollar coffee scores high risk. The job emits a fraud score downstream, where it either approves, declines, or triggers a one-time verification, all before the payment network finishes the authorization.
All of this happens per event, continuously, across millions of cards. There is no nightly job. The moment a charge happens, the answer is already on its way back, which is the whole point of processing the stream instead of the batch.
Apache Kafka and Kafka Streams
Kafka is the durable event log most streaming pipelines read from, and Kafka Streams is a library for doing the processing inside your own service.
Netflix
Runs thousands of Apache Flink jobs to power real-time recommendations, A/B test metrics, and operational alerting across its playback events.
Uber
Uses Flink and Kafka to compute surge pricing, match riders to drivers, and detect fraud as trip and payment events stream in.
Amazon Kinesis
AWS managed service for ingesting and processing streaming data at scale, often paired with Lambda or Managed Service for Apache Flink.
Batch processing collects data over a window and processes the whole set at once on a schedule, accepting latency of minutes to hours. Stream processing handles each event the moment it arrives, producing results in milliseconds to seconds. Batch is simpler and cheaper when you can wait; streaming is required when an answer loses value quickly, like fraud blocking or live alerts.
It means each event affects the result exactly one time even if workers crash and events get replayed. Processors achieve this with periodic checkpoints of their state plus replayable input offsets from Kafka, so on restart they resume from a known-good point without losing or double counting any event. It is harder to guarantee when results are written to external systems that cannot roll back.
Windows turn an infinite stream into finite groups you can aggregate. A tumbling window cuts time into non-overlapping buckets like every five minutes. A sliding window overlaps, recomputing a one-minute total every ten seconds. A session window groups a user's activity until they go idle. Windows are how you answer questions like how many errors occurred in the last minute.
Event time is the timestamp inside the event, when the thing actually happened. Processing time is the clock reading when your system received it. They differ because of network delays and retries, so events arrive out of order. Using event time, combined with watermarks that estimate when late data has stopped arriving, gives correct windowed results regardless of when events show up.
Apache Flink is the most capable open-source engine for stateful, exactly-once processing. Kafka Streams is a lightweight library that runs inside your own application. Apache Spark Structured Streaming brings micro-batch streaming to the Spark ecosystem. On the cloud, Amazon Kinesis and Google Cloud Dataflow offer managed options. Apache Kafka is almost always the event log underneath them.
Design a Real-Time Analytics Dashboard
Design a real-time analytics system - stream processing, WebSocket-powered dashboards, aggregation pipelines, and windowed computations at scale
Stateless Stream Processing
Transform events independently, filtering, mapping, and routing without maintaining state
Stateful Stream Processing
Maintaining state across events, aggregations, joins, and pattern detection in streaming data
Exactly-Once Semantics
Processing every event exactly once despite failures, the holy grail of stream processing
Kappa Architecture
Everything through a single stream processing layer, simplifying Lambda by eliminating the batch layer
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Event Sourcing
Storing every state change as an immutable event instead of just the current state. You can rebuild any past state by replaying events.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
This page explains the idea. The "Stream Processing" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Stream Processing lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Stream Processing in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
