Apache Flink

Apache Flink is a framework for running computations over continuous data streams. A stream is just an unbounded sequence of events: clicks, payments, sensor readings, log lines. Flink ingests those events as they arrive and runs your logic on each one, instead of waiting to collect a big file and process it later.

The key word is stateful. Most useful stream jobs need to remember things: a running count per user, the last 5 minutes of orders, a join between two streams. Flink stores that memory, called state, inside the job itself and manages it for you. State can grow into terabytes and Flink keeps it consistent.

Flink also handles batch data, but it does so by modeling a finite dataset as a stream that happens to have an end. This is the opposite of older systems like Spark, which started as batch and bolted streaming on top by chopping data into tiny micro-batches. Flink is streaming first.

You write a job as a graph of operators: sources read events, transform operators map and filter and aggregate, and sinks write results out. Flink turns that graph into parallel tasks and spreads them across worker processes called TaskManagers, coordinated by a JobManager. Each operator instance handles a slice of the data, partitioned by a key such as user ID.

State lives next to each operator, usually on local disk through an embedded RocksDB store, so reads and writes stay fast. To survive failures, Flink runs the Chandy-Lamport algorithm: it injects markers called barriers into the stream, and when a barrier flows through every operator, Flink writes a consistent snapshot of all state to durable storage like S3 or HDFS. These snapshots are called checkpoints and run every few seconds.

When a machine dies, Flink rolls every operator back to the last completed checkpoint and replays the events that came after it. Because the source position and the state are restored together, each event affects the result exactly once. This is the exactly-once guarantee, and it is the main reason teams pick Flink over simpler tools.

Flink also understands event time, the timestamp baked into each event, separate from when it was processed. Watermarks tell Flink how far event time has progressed so it can correctly close time windows even when events arrive late or out of order, which is normal in real networks.

Reach for Flink when you need low latency on continuous data and the logic depends on accumulated state: real-time fraud scoring, live dashboards, alerting on metrics, joining a stream of orders with a stream of payments, or continuous ETL into a warehouse. Sub-second results and correct math under failure are where it earns its keep.

The cost is operational weight. Flink is a distributed stateful system, so you run a cluster, tune memory and RocksDB, size checkpoint storage, and plan for state that can reach terabytes. Restoring a huge job from a checkpoint takes time, and a badly tuned job can stall under backpressure when a slow sink pushes the slowness upstream.

If your problem is a once-a-day report over files, plain batch with Spark or even SQL is simpler and cheaper. If you just need to move events between systems with light transforms, Kafka Streams or a managed service may be enough. Flink shines when correctness, latency, and large state all matter at the same time.

Imagine a payments company that wants to flag a card the moment it is used in two countries within 60 seconds. A Flink job reads the transaction stream from Kafka keyed by card number. For each card it keeps state: the last country and the timestamp of the last swipe.

When a new transaction arrives, the operator compares it against that state. If the country differs and the two events fall inside a 60 second event-time window, it emits an alert to another Kafka topic and the fraud team reacts in real time. The window logic uses watermarks so a transaction delayed by the network still lands in the right window.

If a TaskManager crashes mid-stream, Flink restores every card's state from the last checkpoint and replays the Kafka offsets that were saved in that same checkpoint. No alert is missed and none is duplicated. That combination of per-key state, event-time windows, and exactly-once recovery is exactly what Flink was built for.