Design an ad click aggregator

Counting is the easiest thing a computer does, right up to the moment the count is money. Then a duplicate is fraud, a dropped event is a refund, and an advertiser who cannot reconcile your number against theirs stops spending.

This question is asked at senior level because it has no comfortable answer. Streaming gives you fast numbers that are occasionally wrong. Batch gives you correct numbers hours late. The design is how you offer both without maintaining two truths.

Step 1: Understand the problem#

The word "count" hides four different requirements. Separate them before you draw, because each one pulls the design somewhere else.

What you are building#

Back of the envelope#

Step 2: Propose the high level design#

The API#

{
  "eventId": "c-8f21ab...",
  "type": "click",
  "adId": 88231,
  "eventTime": "2026-08-25T14:59:58.221Z"
}

{ "from": "2026-08-25T14:00Z", "to": "2026-08-25T16:00Z" }

The data model#

The whole system on one whiteboard#

Walking Figure 1:

1. The click is a redirect through your ad server, which is what makes the event yours to count rather than something you have to be told about.
2. The ad server appends to the log and returns immediately. It never waits for anything downstream, and it never aggregates.
3. Events are deduplicated by the click id minted at step 1, before anything counts them.
4. The stream processor groups them into one minute windows by event time.
5. Those windows land in the serving store within about a minute of the click.
6. In parallel, everything on the log is archived raw. This archive, not the serving store, is the source of truth.
7. An hourly batch job reads the archive and computes the same windows exactly, with all the late data that has arrived since.
8. It overwrites the rows the stream already wrote. Same table, same rows, better number.
9. Reconciliation compares your totals against what the advertiser's own system saw, because the disagreement is the thing that actually costs money.
10. The dashboard and the invoice read the same table. A row starts out approximate and becomes final, and a flag says which.

Edge 8 is the design. The batch job does not write to a different table for a different audience. It overwrites the same rows the stream produced, so a number on the dashboard gets more accurate over the following hours and then stops changing. There is one table and one truth, with a finalised flag saying which rows are safe to bill against.

Step 3: Design deep dive#

Event time is not arrival time#

The click happened at 14:59 on a phone with a poor connection. It reached you at 15:04. Which minute does it belong to?

Event time, always. Bucketing by arrival time means a network delay silently moves revenue between advertisers, and it means replaying the log produces different answers than the original run, which destroys any hope of reconciliation.

Choosing event time creates the problem this whole section is about: you never know when a window is complete, because there might always be one more straggler.

Drag the slider from zero to the end. At zero you publish the moment the window closes and lose three events. At thirty you lose none and every number is thirty seconds older. There is no setting that gives both, and finding one is not an engineering problem you have failed to solve.

So do not choose. Emit early with what you have, keep the window open, and emit again if stragglers arrive. The dashboard shows the early number and corrects itself. Billing waits for the batch job, which reads the archive long after every straggler has landed.

Counting each click exactly once#

Every layer here delivers at least once. The ad server retries on a timeout, the log redelivers on a consumer restart, and the stream processor reprocesses a window after a crash. Without care, a single click becomes three.

Three ideas are doing all the work there, and they are worth naming separately because each one solves a different failure:

Event ids created at the edge. Anything generated later cannot tell a retry from a second real click.

Deduplication scoped to the window. Remembering every id ever seen is not affordable. Remembering ids within a window is, and a duplicate arriving after its window has closed is rare enough to be handled by the batch job.

Writes that set rather than increment. An increment is not idempotent, so reprocessing doubles it. Writing the computed total for a window keyed by that window makes reprocessing free, which in turn makes crash recovery boring.

Two pipelines, one number#

Partitioning the stream#

Break it#

Trade-offs#

Interview replay#

Checkpoint#