Ad Click Aggregator System Design Interview: Counting a Firehose Without Double-Charging Advertisers
A large ad network emits an event firehose measured in millions of impressions and clicks per second. Advertisers want a dashboard that updates within seconds, but they are billed on those same numbers, so a click counted twice is money charged twice. The whole design lives in the gap between fast approximate counts and slow exact counts. All throughput figures here are estimates unless a cited source says otherwise.
An ad click aggregator ingests a massive stream of click and impression events, groups them into per ad, per time bucket counts, and serves two audiences from the same data: advertisers watching a near real time dashboard and a billing system that needs numbers it can defend. The core of the design is a durable log (Kafka) that absorbs the firehose and partitions events by ad id, a stream processor (Flink or Spark Structured Streaming) that runs windowed aggregation over event time, and an OLAP store (Druid, Pinot, or ClickHouse) that holds pre aggregated rollups and answers dashboard queries in well under a second. The hard parts are all about correctness under failure. Events arrive out of order and late, so you need watermarks and a grace period to decide when a minute is done. A click must be counted exactly once, so every event carries a stable id and gets deduplicated. A viral ad creates a hot partition that one consumer cannot keep up with. And because streaming counts can drift, most real systems keep a batch reconciliation path (or a replayable single stream) that produces the authoritative billing number a few hours later. The interview is really a tour of stream processing done under a billing constraint.
Asked at: Asked at Meta, Google, Amazon, Uber, TikTok, LinkedIn, and most ad tech and data infrastructure teams. It is a staple of senior and staff loops because it forces a candidate to reason about streaming correctness rather than just drawing boxes.
Why this question is asked
Interviewers like this problem because it separates people who have only read about stream processing from people who understand what actually breaks. Almost any candidate can say 'Kafka into Flink into a database.' The signal comes from the follow ups. What happens when an event arrives ten minutes late? How do you know a one minute window is complete if you cannot trust wall clock time? How do you avoid counting the same click twice when a processor crashes and replays? What do you do when one ad goes viral and its partition falls behind while every other partition is idle? The billing angle makes correctness non negotiable, so hand waving gets exposed quickly. It also has natural depth: you can spend the whole hour on windowing and watermarks, or pivot into the Lambda versus Kappa debate, or dig into the OLAP serving layer. Few problems test event time reasoning, idempotency, hot key handling, and the accuracy versus latency tradeoff all at once.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Ingest click and impression events from ad serving endpoints across many regions, on the order of millions of events per second at peak.
- Aggregate counts per ad id per time bucket (per minute, rolled up to hourly and daily) with support for slicing by dimensions like campaign, advertiser, country, and device.
- Serve a near real time dashboard where an advertiser sees clicks and spend updating within a few seconds of the click.
- Produce accurate, reconciled counts that the billing system can charge against, even if that number lags real time by minutes or hours.
- Count each click exactly once so an advertiser is never charged twice for the same event.
- Handle out of order and late arriving events correctly, attributing them to the time bucket they actually happened in.
- Support top N queries such as the highest converting ads in the last hour, and time range queries over arbitrary windows.
- Filter fraudulent and bot traffic (click spam) so it does not inflate billed counts.
- Allow reprocessing of a time range if a bug is found in the aggregation logic, without corrupting already served data.
Non-functional requirements
- High ingest availability. Dropping the endpoint should never block ad serving, so the collector must be fire and forget and the log must absorb bursts.
- Dashboard query latency in the low hundreds of milliseconds at p99 for typical rollup queries.
- End to end freshness for the real time path measured in seconds to low tens of seconds.
- Correctness for the billing path. The reconciled number must be exact and auditable, not approximate.
- Horizontal scalability across ingestion, processing, and storage as event volume grows.
- Durability. No acknowledged event may be lost, since a lost click is lost revenue.
- Fault tolerance. A processor crash must not double count or drop counts on recovery.
- Cost efficiency. Storing every raw event forever is expensive, so raw retention is bounded while rollups are kept long term.
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Impression firehose (peak)
~3M events/sec
Estimate for a large network. Assume 100B impressions/day. 100e9 / 86400 = 1.16M/sec average. Apply a 2.5x to 3x peak factor for prime time. Derived, not a published Meta or Google figure.
Click events (peak)
~35K events/sec
Derived from the impression estimate at ~1% click through rate. 100B impressions/day * 1% = 1B clicks/day, /86400 = ~11.6K/sec average, ~35K/sec at 3x peak. Clicks are the billed events and are far rarer than impressions.
Ingest bandwidth
~600 MB/sec
Derived. 3M events/sec * ~200 bytes per serialized event (ids, timestamp, ad id, geo, device, user agent hash) = 600 MB/sec into Kafka before replication. With replication factor 3 the disk write rate is roughly 1.8 GB/sec across the cluster.
Aggregate rows per day (per minute grain)
~14B rows/day
Derived and pessimistic. Assume 10M ads active in a day and 1440 minute buckets: 10e6 * 1440 = 14.4B rows. Real systems are far smaller because most ads are idle in most minutes, so sparse storage plus rollup collapses this by one to two orders of magnitude.
Dashboard query latency target
<200 ms p99
A design target, not a measured number. Achieved by querying pre aggregated rollups in a columnar OLAP store rather than scanning raw events, so a one week trend for one advertiser touches thousands of rolled up rows, not billions of raw ones.
Raw event retention vs rollup retention
~3 to 7 days raw, 13+ months rollup
Estimate driven by cost. Raw events are kept only long enough to reprocess and reconcile (Kafka retention plus a data lake copy). Minute and hour rollups are cheap and kept for year over year reporting and audits.
High-level architecture
An ad event starts at the ad server. When a user clicks, the browser hits a lightweight collector endpoint (often a redirect service that logs the click, then 302s the user to the advertiser landing page). The collector does almost no work: it stamps the event with a server side receive time, validates the payload, attaches a stable event id, and appends it to a Kafka topic partitioned by ad id. Kafka is the buffer that decouples a spiky, always on firehose from the processing layer, and it is also the replay log that makes reprocessing possible. A stream processor, typically Flink, consumes the topic, deduplicates events using their ids against keyed state, and runs tumbling windows (one minute is common) keyed by ad id and dimensions. It uses event time, not processing time, and advances watermarks so it knows when a window can be considered complete despite out of order arrivals. When a window closes, the processor emits a compact aggregate row (ad id, minute, click count, impression count, spend) and writes it to a real time OLAP store such as Apache Druid or Apache Pinot, using an upsert keyed on ad id plus time bucket so retries and late corrections overwrite rather than duplicate. The advertiser dashboard queries that OLAP store directly and gets sub second responses because it reads pre aggregated rollups. In parallel, the raw events also land in a data lake (S3, Hive) so a batch job can recompute the exact numbers later. Billing reads from the reconciled batch output, or from the OLAP store once the streaming counts have stabilized, never from the raw provisional counts. Fraud filtering sits either inline in the stream (fast heuristics) or as a separate scoring job that adjusts counts before billing.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Event collector and redirect service
A stateless, horizontally scaled HTTP tier that receives click and impression beacons and writes them to Kafka. For clicks it usually issues a 302 redirect so the user reaches the landing page immediately while the event is logged asynchronously. It must be fire and forget: it acknowledges fast and never blocks on downstream processing, because ad serving latency directly affects revenue.
Kafka ingestion log
The durable, partitioned commit log that absorbs the firehose and orders events within a partition. Partitioning by ad id keeps all events for one ad on one partition so aggregation state is local, and it is what lets Pinot or Druid upsert correctly. Its retention window doubles as the replay buffer for reprocessing and for the batch reconciliation path.
Stream aggregation job (Flink or Spark Structured Streaming)
The heart of the real time path. It reads from Kafka, deduplicates by event id using keyed state, assigns event time, runs windowed aggregation with watermarks, and emits per ad, per bucket counts. It checkpoints its state and Kafka offsets together so a crash and restart resumes without double counting.
Real time OLAP serving store (Druid, Pinot, or ClickHouse)
A columnar store built for fast group by and time range scans over aggregates. It ingests the streaming rollups and makes them queryable within seconds, and it uses upsert on the ad id plus time bucket key so late corrections replace earlier values. This is what the advertiser dashboard actually queries.
Batch reconciliation pipeline
A periodic job (Spark or a warehouse SQL job) that reads the raw events from the data lake and computes exact counts for closed time ranges. It corrects for anything the streaming layer got wrong: dropped watermarks, extreme late arrivals, or fraud rescoring. Its output is the source of truth for billing.
Fraud and click spam filter
A scoring stage that flags bot traffic, duplicate rapid clicks from one device, and click farms. Simple heuristics (rate per device, impossible click through rates) can run inline in the stream, while heavier machine learning scoring runs as a separate job whose verdicts adjust the billed counts before invoicing.
Query and billing API
A service layer that fronts the OLAP store for dashboard reads and exposes the reconciled counts to the billing system. It enforces per advertiser access control, applies currency and spend calculations, and clearly separates provisional real time numbers from finalized billed numbers so nobody is charged off a draft count.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
raw_ad_eventsevent_idevent_typead_idevent_timereceive_timeAppend only log of every click and impression, keyed by a globally unique event_id used for deduplication. event_time is when it happened, receive_time is when the collector got it; the gap drives late data handling. Also carries campaign_id, advertiser_id, geo, device, and a hashed user identifier. Kept only for a short retention window plus a data lake copy.
click_dedup_stateevent_idad_idfirst_seen_atwindow_keyConceptually the keyed state the stream processor holds to reject duplicate event_ids. In Flink this lives in RocksDB backed state with a TTL roughly equal to the window plus allowed lateness, not a queryable table. Bounded TTL keeps it from growing without limit.
ad_counts_minutead_idminute_bucketclick_countimpression_countspendThe primary rollup emitted by the stream job, one row per ad per minute. Written with upsert on (ad_id, minute_bucket) so retries and late corrections overwrite. Additional dimension columns (campaign, geo, device) let the OLAP store answer sliced queries. This is what the dashboard reads.
ad_counts_hourlyad_idhour_bucketclick_countimpression_countspendCoarser rollup derived from the minute table, kept far longer for trend and year over year queries. Rolling up from minutes to hours reduces row count by up to 60x and keeps historical queries cheap.
billing_reconciled_countsad_iddate_bucketbillable_clicksfraud_adjusted_clicksfinalized_atOutput of the batch reconciliation and fraud scoring, the authoritative number for invoicing. finalized_at marks when a period was closed and can no longer change. Billing reads only rows with a finalized_at set, never the provisional streaming counts.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Event time versus processing time, and why watermarks are the whole game
The naive design buckets an event by the wall clock time the processor sees it. That is processing time, and it is wrong for billing. A click that happened at 12:00:59 but arrives at 12:01:03 because of a mobile network delay would land in the wrong minute, and the 12:00 total would never be right. So the aggregation keys on event time, the timestamp stamped when the click actually occurred. The problem event time creates is that you can never be sure a minute is finished, because a straggler could always show up. Watermarks solve this. A watermark of time T is an assertion from the stream that it does not expect any more events with timestamp at or before T. When the watermark passes the end of a window, the processor fires that window and emits its count. Flink generates watermarks from the observed event timestamps with a configured out of orderness bound, for example 'assume events can be up to 30 seconds late.' Choosing that bound is the central tradeoff: a large bound means windows fire late and the dashboard lags, a small bound means more events arrive after the window closed and get treated as late data. This is why the interviewer keeps asking about late events. It is the one knob that trades freshness against completeness.
Handling late and out of order events without lying to the advertiser
Even with a sensible watermark, some events arrive after their window has fired. You have three options and real systems use a mix. First, drop them: acceptable for a dashboard where a handful of stragglers do not matter, unacceptable for billing. Second, allow a grace period (Flink calls it allowed lateness) during which a late event reopens its window, recomputes the count, and emits an updated result. Because the OLAP store upserts on (ad_id, minute_bucket), that update simply overwrites the old value and the dashboard corrects itself. Third, route very late events to a side output and fold them in through the batch reconciliation path rather than the stream. The clean mental model is that the streaming number is always provisional and self correcting for a while, and the batch number is final. This is exactly why you keep the raw events: the batch job can attribute a click that showed up hours late to the correct original minute, something the stream gave up on once its state TTL expired.
Exactly once and deduplication as a billing requirement
Double counting a click means double charging an advertiser, so exactly once is not a nice to have here. Two different duplication sources exist. The first is duplicate events at the source: retries from the client beacon, or the same click logged twice. You defend against this by giving every event a stable id at creation and deduplicating against keyed state in the processor, so the second copy of an id is dropped. The second source is reprocessing after failure: when a Flink job crashes and restarts, it replays from the last checkpoint, so some events get processed twice. Flink handles this with checkpoint barriers that snapshot operator state and Kafka offsets together, and with transactional (two phase commit) sinks so partial output from a failed run is never committed. Uber's ad platform combines both ideas: Flink and Kafka run in exactly once mode with read_committed consumers, checkpoints commit transactionally, and every aggregated record carries a UUID that downstream systems (Pinot via upsert, Hive via dedup, the ad budget service via an idempotency key) use to reject any duplicate that still slips through. Layering source dedup, transactional processing, and idempotent sink writes is what makes the end to end count trustworthy.
Hot partitions when an ad goes viral
Partitioning by ad id gives you local aggregation state and clean upserts, but it creates a skew risk. If one ad goes viral, its partition receives orders of magnitude more events than the others, and the single consumer and single aggregation key for that ad becomes a bottleneck while the rest of the cluster sits idle. The fix is to split the hot key. Instead of keying on ad_id, you key on ad_id plus a random salt in a small range, so a viral ad spreads across, say, 16 sub keys and 16 parallel aggregators, then you sum the 16 partial counts in a second, lighter aggregation step. The salt range can be fixed for everyone or applied adaptively only to keys detected as hot. Kafka side, you can also over partition so no single partition is too coarse, and use a partitioner that spreads a hot ad rather than pinning it. The interview point is recognizing that the natural partition key is also the natural hot spot, and that two stage aggregation (partials then merge) is the standard escape hatch.
Lambda versus Kappa, the accuracy and latency reconciliation
The billing constraint pushes you toward two answers for the same question: a fast approximate count now, and a slow exact count later. Lambda architecture makes this explicit with two code paths, a speed layer (the stream job serving the dashboard) and a batch layer (recompute from raw events for billing), with a serving layer that merges them. It works, but Jay Kreps' well known critique is that maintaining two implementations of the same aggregation logic in two frameworks is a bug factory, because the batch and streaming results drift whenever the two code paths diverge. Kappa architecture answers this by keeping only the streaming path and treating the Kafka log itself as the batch layer: to reprocess, you replay the topic from the beginning through a new version of the same job, so there is one implementation and one framework. Kappa is cleaner when your retention and reprocessing throughput can cover the historical window. In practice many ad systems still keep a real batch reconciliation because raw retention in Kafka is expensive and auditors want an independent recomputation. The honest interview answer names both, explains the drift problem Lambda has and the retention and replay cost Kappa has, and picks based on how long you must be able to reprocess and how independent the billing recomputation needs to be.
The OLAP serving layer: rollups, pre aggregation, and fast dashboards
The dashboard cannot scan billions of raw events per query, so the serving store holds pre aggregated data in a columnar format tuned for time series group by. Apache Druid rolls up at ingestion time by collapsing rows that share the same dimensions and timestamp bucket into one, which can cut row counts by orders of magnitude and is the reason a trend query stays fast. Apache Pinot takes a similar columnar approach and adds native upsert, which Uber contributed and uses precisely so that reprocessed or late corrected aggregates replace the earlier version rather than pile up as duplicates; Pinot requires the input stream to be partitioned by the primary key for upsert to work, which lines up with partitioning Kafka by ad id. ClickHouse is a common alternative with materialized views doing the rollup. Whichever you pick, the pattern is the same: write compact aggregates keyed by ad id and time bucket, keep multiple rollup granularities (minute, hour, day) so each query hits the coarsest table that answers it, and lean on the store's upsert or replace semantics so corrections are idempotent. The store is optimized for reads and append or upsert of aggregates, never for holding the full raw firehose.
Fraud and click spam filtering before the money is charged
A meaningful fraction of ad traffic is not real: bots, click farms, and accidental or malicious repeated clicks. If those inflate the billed count, advertisers are overcharged and eventually leave. Filtering has two speeds. Cheap inline heuristics run in the stream and catch the obvious cases: many clicks from one device or IP in a short window, a click through rate that is physically implausible, clicks with no preceding impression, or user agents known to be automated. Heavier detection runs offline as a scoring model over sessions and device graphs, and its verdicts feed the batch reconciliation so fraudulent clicks are subtracted before invoicing. The design consequence is that the billed number is not just deduplicated, it is fraud adjusted, and that adjustment usually happens on the batch path where you can afford the extra latency and the richer features. Keeping fraud scoring separate from the fast aggregation also means you can retrain and rescore historical windows without touching the hot ingestion path.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Event time versus processing time
Processing time is simpler and always makes progress, but it attributes events to the wrong buckets under network delay, which is unacceptable when those buckets are billed. Event time with watermarks is more complex and can stall if a source goes quiet, but it is the only way to get correct per minute counts, so it wins for anything touching billing.
Lambda (dual path) versus Kappa (single stream) architecture
Lambda gives you an independent exact recomputation for billing at the cost of maintaining two code paths that drift. Kappa keeps one implementation and reprocesses by replaying the log, which is cleaner but needs long Kafka retention and enough replay throughput, and gives auditors less independence. Pick Lambda when the billing recomputation must be independent and auditable, Kappa when operational simplicity and a single codebase matter more.
Small watermark delay versus large allowed lateness
A short out of orderness bound fires windows quickly so the dashboard is fresh, but more events end up late and need correction. A long bound and generous grace period capture almost everything on the first pass but make the dashboard lag. Since the dashboard is provisional and billing is reconciled anyway, most systems favor freshness on the stream and lean on batch for the stragglers.
Partition by ad id versus salted keys
Plain ad id partitioning keeps aggregation state local and makes OLAP upserts trivial, but a viral ad creates a hot partition that one consumer cannot drain. Salting the hot key spreads load across parallel aggregators and requires a second merge step. Use plain keys as the default and apply salting only to detected hot keys so you do not pay the merge cost everywhere.
Pre aggregated rollups versus storing raw events for queries
Serving queries from raw events gives maximum flexibility but cannot meet sub second latency at this volume. Pre aggregating into minute and hour rollups makes dashboards fast and cheap at the cost of losing per event detail and fixing your dimension set in advance. Keep raw events only in the lake for reprocessing, and serve everything user facing from rollups.
Bill from real time streaming counts versus reconciled batch counts
Billing off the streaming number would be immediate but risks charging for counts that later get corrected down by late data or fraud rescoring. Billing off the reconciled batch number lags by hours but is exact and defensible. The standard answer is to show streaming numbers as provisional in the UI and charge only the finalized reconciled numbers.
How Ad Click Aggregator actually does it
Uber published a detailed account of its ad event processing platform built on Apache Flink, Kafka, and Pinot. It runs Flink and Kafka in exactly once mode with read_committed consumers and two phase commit sinks, checkpoints every two minutes (which it calls out as the main reason the system is near real time rather than instant), and attaches a record UUID to every aggregate so Pinot upsert, Hive dedup, and the ad budget service can each reject duplicates. It uses one minute tumbling windows keyed by ad identifier, stores events in Docstore for order attribution, and processes hundreds of millions of ad events per week. The broader design vocabulary comes from Apache Flink's own documentation on event time and watermark generation, from Jay Kreps' Questioning the Lambda Architecture which framed the Kappa alternative, and from the serving layer docs for Apache Druid (ingestion time rollup) and Apache Pinot (stream ingestion with upsert, which Uber contributed). These are the primary references a candidate should anchor on rather than generic blog summaries.
Sources
- Uber Engineering: Real-Time Exactly-Once Ad Event Processing with Flink, Kafka, and Pinot
- Jay Kreps: Questioning the Lambda Architecture (O'Reilly)
- Apache Flink: Generating Watermarks (event time documentation)
- Apache Pinot: Stream Ingestion with Upsert
- Apache Druid: Data Rollup (ingestion-time pre-aggregation)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Ad Click Aggregator.