Walkthrough: Ad Click Aggregator

An ad click aggregator ingests a firehose of click events and lets advertisers ask "how many clicks did my ad get, per minute / day / week?" It is a pipeline question — data flows through transformations — and it's also a billing question, which is what makes correctness interesting: advertisers pay per click, so duplicates and drops are money.

Step 1: Requirements

Functional:

1. Record ad clicks (and redirect the user to the advertiser's site).
2. Advertisers query click counts: near-real-time at minute granularity for recent data, day/week granularity for history.
3. Counts must be accurate enough to bill on — no double-counting a click.

Non-functional:

• Scale: ~10k clicks/sec peak (~100M clicks/day).
• Query latency for advertisers: sub-second on aggregates.
• Freshness: recent data visible within about a minute.

Step 2: Data flow first

Narrate before drawing: click arrives → validate and dedupe → append to a durable stream → aggregate over time windows → write aggregates to an analytics store → advertisers query. Raw events are also archived, which becomes important later.

Step 3: High-level design

Ingestion. The ad's link points at our /click endpoint, which logs the event and returns a server-side redirect (302) to the advertiser — tracking is guaranteed even when browsers block scripts. The click processor validates the event and appends it to Kafka (a durable, replayable log), partitioned by AdId so each ad's events stay ordered on one partition. Kafka absorbs 10k/sec trivially (~1M msgs/sec per broker) and buffers downstream hiccups.

Aggregation. Choose your tier deliberately:

• Good — batch: land raw events in wide-column storage (Cassandra suits the write rate), run Spark every few minutes to compute per-minute counts. Simple, correct, but freshness is "minutes behind". A mid-level candidate can land here and pass.
• Great — streaming: Flink (a stream processor) consumes Kafka and maintains windowed aggregations — e.g., a count per AdId per 1-minute event-time window (bucketing by when the click happened, not when it arrived). Freshness drops to seconds.

Serving. Minute-level aggregates flow into an OLAP store (a column-oriented analytics database — Redshift/Snowflake-style — built for scan-and-group queries, the opposite shape of an OLTP row store). A nightly job pre-aggregates minute rows into day/week rollups so historical queries stay sub-second without storing minute rows forever. Raw events also land in S3 as a data lake — the immutable record.

Deep dive 1: Dedupe — the billing guarantee

Users double-click; browsers retry. The standard mechanism: when the ad is served, it carries a signed impression ID (unique per ad view, HMAC-signed so it can't be forged). At click time the processor verifies the signature and does an atomic check-and-set on the impression ID in Redis: seen → drop; new → admit to Kafka. Dedupe happens before the stream, so everything downstream can trust the events.

Deep dive 2: Exactly-once, watermarks, and late events

Inside Flink, expect the semantics probe. Flink offers exactly-once state processing via checkpointing — after a crash it rewinds to a consistent snapshot and replays Kafka from the recorded offsets, so no window double-counts. Be precise: this is exactly-once within the pipeline; end-to-end still depends on idempotent writes to the OLAP store (e.g., keyed upserts per ad+window).

Event time needs watermarks: a watermark is the stream's declaration "I believe all events up to time T have arrived", which is what lets a 12:00–12:01 window ever close despite network delays. Events arriving after the watermark are late — either fold them in via an allowed-lateness update to the emitted window, or route them to a side output and let reconciliation catch them.

Deep dive 3: Hot partitions — the viral ad

Partitioning by AdId means one viral ad funnels its entire spike into a single partition, and one consumer melts. Fix: salt the key — write to AdId:0..N (random suffix), spreading the ad across N partitions, then strip the salt at aggregation (the window operator sums the N sub-counts). Detect hot ads dynamically and apply salting only to them.

Deep dive 4: Lambda vs kappa

Why keep raw events in S3 if Flink already aggregates? Streaming results can drift — bugs, late events beyond allowed lateness, misconfigured windows. The lambda architecture runs a nightly batch job (Spark over the S3 raw events) recomputing the same aggregates and reconciling: the batch result overwrites the streaming result, correcting drift. You bill from the corrected numbers. The kappa alternative says: skip batch entirely — Kafka is replayable, so re-run the stream job over history when corrections are needed. Kappa is one codebase; lambda is the belt-and-suspenders answer for money. For billing, pick lambda and say why.

Wrap-up

Recap: signed impression IDs dedupe at the door; Kafka partitioned by AdId (salted when hot) buffers 10k clicks/sec; Flink event-time windows with watermarks deliver minute-fresh counts; an OLAP store with nightly rollups serves advertiser queries sub-second; and nightly batch reconciliation over the S3 raw events keeps billing honest. With more time: per-advertiser query auth, and cost controls on OLAP storage granularity.