Event sourcing, CDC and the log

A support ticket arrives: "My wallet balance is ₹400 less than it should be." The engineer opens the database. The wallets row says balance = 1150. That is all it says. It does not know how it got there, what it was yesterday, or whether a refund was applied twice. The one thing support needs, the history, was overwritten one UPDATE at a time.

An accountant would never keep books this way. A ledger only ever has entries added to it, and the balance is what you get by adding them up. This chapter is about building software the same way, and about getting that history out of databases that were not built like that.

The log underneath everything#

You have already met the same structure four times on this site. A database writes every change to its write ahead log before touching its tables. Raft keeps replicas in step by agreeing on a log. Replicas copy the primary by replaying its log. A Kafka topic is a log you can read.

In each case the log is the truth and the tables are a convenience built from it. Jay Kreps, one of Kafka's creators, made this argument in a 2013 essay, "The Log", and it changes how you see data: a table is just the latest value of every key, which is a cache of the log.

Event sourcing#

Event sourcing makes that explicit in your application. Instead of storing the current state and updating it, you store every change as an event, an immutable fact in the past tense (MoneyAdded, PaymentMade), and derive the current state by applying the events in order.

Four things come for free that a CRUD table cannot give you:

A complete audit trail. Every rupee is accounted for, in order, forever. For money this is often a regulatory requirement anyway.

Time travel. "What was the balance before the bill?" is a replay up to that event.

New views of old data. A fraud team that arrives a year later can build its report from the very first event, as if it had been there all along.

Debugging by replay. Copy the events of the account that went wrong, replay them through the code, and watch the bug happen.

In TypeScript, the event types and the fold that derives the state look like this. The union of object types (a discriminated union) lets the compiler check that every event type is handled.

type WalletEvent =
  | { type: 'MoneyAdded'; amount: number }
  | { type: 'PaymentMade'; amount: number; to: string }
  | { type: 'PaymentRefunded'; amount: number }
 
// State is never stored as truth. It is what you get by applying every event in order.
function apply(balance: number, e: WalletEvent): number {
  switch (e.type) {
    case 'MoneyAdded':
    case 'PaymentRefunded':
      return balance + e.amount
    case 'PaymentMade':
      return balance - e.amount
  }
}
 
const balance = events.reduce(apply, 0)

Commands, events and views (CQRS)#

Event sourcing usually comes with CQRS, command query responsibility segregation: the side that accepts changes and the side that answers questions are built separately.

1. A command asks for a change. It can be refused.
2. The service loads the stream's events (or a snapshot plus recent events), checks the rules, and appends one or more events. Events are never refused: they already happened.
3. Events are published, typically through the outbox or by the event store itself.
4. Projections consume events and keep read models up to date, each shaped for its queries.

The read models lag the event store by milliseconds to seconds, so the consistency models chapter applies: give users read your writes on the views they just changed, for example by returning the new balance in the command's response.

Two writers, one stream#

Two payments from the same wallet at the same moment must not both pass a balance check made against the same old state. Event stores solve this with optimistic concurrency: you append "as version 8", and the append fails if someone else already wrote version 8. Then you reload and check again. In Postgres it is a unique constraint:

CREATE TABLE wallet_events (
  wallet_id  bigint      NOT NULL,
  version    int         NOT NULL,   -- 1, 2, 3 ... per wallet
  type       text        NOT NULL,
  data       jsonb       NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (wallet_id, version)   -- a second writer of version 8 gets a conflict
);

Long streams are slow to replay, so services save a snapshot of the state every few hundred events and replay only what came after it. The snapshot is a cache. The events stay the truth.

What it costs#

Event sourcing is powerful and expensive, and the costs show up late.

Events are forever, so their shape is too. Change what a PaymentMade contains and you must still read every old version, for years. Teams version events and convert old ones as they are read (called upcasting).

Deleting personal data is hard. An append-only log fights the right to erasure under GDPR and India's Digital Personal Data Protection Act, 2023. The usual answer is crypto-shredding: encrypt each person's personal fields with their own key, and delete the key when they ask.

Views are eventually consistent, and every new view is code to run, monitor and rebuild.

It is the wrong tool for most data. A settings page, a product catalogue or a blog does not need its history replayed. Use event sourcing where the history is the product: ledgers, orders, bookings, workflows that are audited.

CDC: the log of a database you already have#

Most systems are not event sourced, and still need their changes as a stream: to update a search index, invalidate a cache, feed analytics, or move data out of a monolith. Change data capture (CDC) gets that stream from the database's own log, without touching the application.

CDC removes the dual write problem (writing to the database and then to Kafka, and crashing in between) because there is only one write. Kafka's log compaction, which keeps only the latest message per key, can then hold a full, current copy of the table as a topic.

Interview replay#

Checkpoint#