We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A distributed stream processing framework that handles both real-time streams and batch data with exactly-once guarantees. Used by Alibaba, Netflix, and Uber at massive scale.
In short
Apache Flink is an open source distributed engine for stateful processing of data streams. It treats batch data as a bounded stream, processes records one at a time as they arrive with millisecond latency, and keeps large amounts of application state with exactly-once correctness even when machines crash.
In short
Apache Flink is an open source distributed engine for stateful processing of data streams. It treats batch data as a bounded stream, processes records one at a time as they arrive with millisecond latency, and keeps large amounts of application state with exactly-once correctness even when machines crash.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Apache Flink across a run of lessons, not one page. These 4 alone are about 131 minutes of step-by-step reading, every one with a quiz.
Save system state at known-good points to enable fast recovery after failures.
Maintaining state across events, aggregations, joins, and pattern detection in streaming data.
Process large volumes of data in scheduled jobs, the workhorse of data pipelines.
Process data as it arrives, real-time analytics, fraud detection, and event-driven architectures.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Checkpointingpay once,
yours for life
Apache Flink is a framework for running computations over continuous data streams. A stream is just an unbounded sequence of events: clicks, payments, sensor readings, log lines. Flink ingests those events as they arrive and runs your logic on each one, instead of waiting to collect a big file and process it later.
The key word is stateful. Most useful stream jobs need to remember things: a running count per user, the last 5 minutes of orders, a join between two streams. Flink stores that memory, called state, inside the job itself and manages it for you. State can grow into terabytes and Flink keeps it consistent.
Flink also handles batch data, but it does so by modeling a finite dataset as a stream that happens to have an end. This is the opposite of older systems like Spark, which started as batch and bolted streaming on top by chopping data into tiny micro-batches. Flink is streaming first.
You write a job as a graph of operators: sources read events, transform operators map and filter and aggregate, and sinks write results out. Flink turns that graph into parallel tasks and spreads them across worker processes called TaskManagers, coordinated by a JobManager. Each operator instance handles a slice of the data, partitioned by a key such as user ID.
State lives next to each operator, either in memory or, for large state, on local disk through an embedded RocksDB store, so reads and writes stay fast. To survive failures, Flink runs the Chandy-Lamport algorithm: it injects markers called barriers into the stream, and when a barrier flows through every operator, Flink writes a consistent snapshot of all state to durable storage like S3 or HDFS. These snapshots are called checkpoints and run every few seconds.
When a machine dies, Flink rolls every operator back to the last completed checkpoint and replays the events that came after it. Because the source position and the state are restored together, each event affects the result exactly once. This is the exactly-once guarantee, and it is the main reason teams pick Flink over simpler tools.
Flink also understands event time, the timestamp baked into each event, separate from when it was processed. Watermarks tell Flink how far event time has progressed so it can correctly close time windows even when events arrive late or out of order, which is normal in real networks.
Reach for Flink when you need low latency on continuous data and the logic depends on accumulated state: real-time fraud scoring, live dashboards, alerting on metrics, joining a stream of orders with a stream of payments, or continuous ETL into a warehouse. Sub-second results and correct math under failure are where it earns its keep.
The cost is operational weight. Flink is a distributed stateful system, so you run a cluster, tune memory and RocksDB, size checkpoint storage, and plan for state that can reach terabytes. Restoring a huge job from a checkpoint takes time, and a badly tuned job can stall under backpressure when a slow sink pushes the slowness upstream.
If your problem is a once-a-day report over files, plain batch with Spark or even SQL is simpler and cheaper. If you just need to move events between systems with light transforms, Kafka Streams or a managed service may be enough. Flink shines when correctness, latency, and large state all matter at the same time.
Imagine a payments company that wants to flag a card the moment it is used in two countries within 60 seconds. A Flink job reads the transaction stream from Kafka keyed by card number. For each card it keeps state: the last country and the timestamp of the last swipe.
When a new transaction arrives, the operator compares it against that state. If the country differs and the two events fall inside a 60 second event-time window, it emits an alert to another Kafka topic and the fraud team reacts in real time. The window logic uses watermarks so a transaction delayed by the network still lands in the right window.
If a TaskManager crashes mid-stream, Flink restores every card's state from the last checkpoint and replays the Kafka offsets that were saved in that same checkpoint. No alert is missed and none is duplicated. That combination of per-key state, event-time windows, and exactly-once recovery is exactly what Flink was built for.
Alibaba
Runs Flink at massive scale for real-time search ranking and recommendations during Singles Day; its Blink fork was merged back into Flink.
Netflix
Uses Flink for real-time event processing across thousands of streaming jobs feeding personalization and operational metrics.
Uber
Powers surge pricing, fraud detection, and the AthenaX SQL platform on Flink to process trip and rider events in real time.
Apache Kafka
Flink's most common source and sink; jobs read events from Kafka topics and write results back with offsets stored in checkpoints.
Spark started as a batch engine and does streaming by splitting data into small micro-batches, which adds latency. Flink is streaming first: it processes each event as it arrives, giving lower latency and true event-at-a-time semantics, and it treats batch as a stream that ends. Spark is often simpler for pure batch reports; Flink wins for low-latency stateful streaming.
It means each input event affects the job's state and results once, even after a crash. Flink achieves this by snapshotting operator state and source positions together in a checkpoint, then on recovery restoring both and replaying only the events after that checkpoint. With supported sinks like Kafka it can extend the guarantee end to end.
Processing time is the clock on the machine when an event is handled. Event time is the timestamp inside the event itself, set when it actually happened. Flink prefers event time and uses watermarks to handle late or out-of-order events, so a transaction delayed by the network still falls in the correct time window.
State is kept local to each operator. By default Flink uses an embedded RocksDB backend that holds state on local disk, which lets a single job manage state into the terabytes. Periodic checkpoints copy that state to durable storage such as S3 or HDFS so it survives failures.
It is heavier than a plain library. You run a cluster of JobManager and TaskManager processes, tune memory and checkpointing, and plan for large state and backpressure. Managed offerings like Amazon Managed Service for Apache Flink or Ververica remove much of that burden if you do not want to run it yourself.
Apache Spark
A unified analytics engine for large-scale data processing with built-in modules for SQL, streaming, ML, and graph computation. Processes data in-memory for speed.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
This page explains the idea. The course covers it step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
See the System Design Masterclass coursecourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Apache Flink in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
