Design a distributed job scheduler

"Send this reminder at 9am." "Rebuild the report every night at 2." "Retry this export in ten minutes." Every product grows a pile of work that has to happen later, at a time, and it has to happen even when the machine that was going to do it has died.

Watch one run go from a schedule to done.

Two promises pull against each other here. Every due run must start, even if a worker or a scanner dies, which pushes you towards running things again. And a run should have its effect once, which pushes you away from running things again. The design is mostly the machinery that lets you keep the first promise without breaking the second.

A few words used all through this page:

• A job is the definition: what to call, with what, and when. A run is one execution of a job at one scheduled time.
• A lease is a claim with an expiry. A worker holding a run's lease is the one allowed to run it, until the lease runs out.
• A fence token is a number that goes up every time a lease is handed out, so storage can tell an old holder from the current one.
• A misfire is a run whose time passed while nothing was able to start it.

Step 1: Understand the problem#

Ask what "exactly once" means to them before you draw anything. The answer decides whether you are building a scheduler or a distributed transaction system, and you want to be building a scheduler.

What you are building, and what you cut#

Back of the envelope#

A small version first. 50 million runs a day over 86,400 seconds is about 580 a second. If 5% of a day's runs are set for 09:00 exactly, that is 2.5 million runs in one minute, about 42,000 a second, more than 70 times the average. The window slider shows what letting those jobs start any time in the next few minutes does. (The back of the envelope chapter covers the method.)

Step 2: Propose the high level design#

The API#

Four calls. The one that matters most is the one teams never think about: what a target receives.

{
  "schedule": "0 2 * * *",
  "timezone": "Asia/Kolkata",
  "target": "https://reports.internal/rebuild",
  "window": "5m",
  "misfire": "run_once",
  "retry": { "max": 5, "backoff": "exponential" }
}

Idempotency-Key: job_88213@2026-10-28T20:30:00Z
X-Fence-Token: 41
{ ...payload }

The data model#

Two tables. jobs is the definition. runs is the timetable and the history, laid out so the question a scanner asks every second, "what in my partition is due now", is one short range read.

The whole system on one whiteboard#

Figure 1 is the picture at the top, laid flat, with retries and the coordination service for partition leases added.

Walking Figure 1:

1. A service creates a job. The API checks the schedule and time zone, and works out the first run in UTC.
2. The job and its first run are written in one transaction. From now on a job always has exactly one future run waiting in the table.
3. Each scanner owns some partitions, through a lease from the coordination service (etcd or ZooKeeper). Every second or so it reads the runs in its partitions that are due and still scheduled.
4. It marks them queued and puts them on the run queue. The queue spreads work over the workers and soaks up the 09:00 spike, so the scanner never waits on a slow target.
5. A worker takes a run from the queue.
6. It claims the run in the job store with one conditional update: running, a 60 second lease, fence plus one, only if nobody holds a live lease. Zero rows changed means someone else has it, and the worker drops the message.
7. It calls the target with the run id and the fence token, heartbeats the lease while it waits, and then, in one write, records the result and creates the job's next run. A failure goes to a retry delay, and a run out of tries goes to the dead list, where someone is told.

Step 3: Design deep dive#

Finding what is due, without one box that must not die#

The whole job of the scheduler is to notice that a time has come. The simple ways to notice either do not scale or quietly depend on one machine staying alive.

The third rung is leader election with leases, applied once per partition instead of once for the whole system. A scanner that loses its lease has to stop reading its partitions at once, and if it does not, nothing bad follows: a run it enqueues twice is claimed once in the store, which is the next deep dive.

Running it once, when a worker might run it twice#

A worker can die, or freeze, at any moment. If the scheduler waits for proof before running a run again, a dead worker means a skipped run. If it does not wait, a frozen worker that wakes up means a run that happened twice.

Try the race. Push the pause past the lease, then turn the fence check on.

So the honest promise is at least once execution with effects that happen once, as long as the target uses the run id. The scheduler cannot do that last part for the target. It can make it easy: one header, stable across retries, documented on day one. The full argument is in exactly-once in practice.

Recurring jobs, and the clock#

A recurring job is where a scheduler earns its keep or quietly loses runs. Three questions come up in every round.

Where does the next run come from? Not from a calendar of every future run. When a run finishes, the same write that records its result creates the next run, worked out from the cron expression in the job's own time zone. A job always has exactly one future run, so editing or pausing it touches one row.

What about runs missed while the scheduler was down? That is the misfire policy, and it is per job, because the right answer depends on what the job does. Try it.

What about time zones? "Every day at 02:00, Asia/Kolkata" is easy, because India has no daylight saving, so every day has exactly one 02:00. "Every day at 02:30, America/New_York" is not. On the night the clocks go forward, 02:00 jumps to 03:00 and 02:30 does not exist. On the night they go back, 01:30 happens twice. Cron implementations disagree about what to do, so a shared scheduler picks a rule and writes it down: a skipped time runs at the first moment after the gap, and a repeated time runs once. Teams in India tend to meet this bug only when they start scheduling for customers abroad, which is exactly when nobody is testing for it.

Break it#

The slow target is the failure that actually happens. The scheduler is healthy, the target is not, and a shared worker pool turns one team's bad day into everyone's. The fixes are the ones in retries and circuit breakers: a bulkhead per target, backoff with jitter, and a breaker that stops calling a target that is down.

Trade-offs#

The level bar#

One moment from this round, answered three ways. Pick the one you think is staff level before the labels appear. (What changes at senior and staff explains the gap.)

Then run the question cold, with a 45 minute timer and no notes, and score the run on the five things this question is really testing.

Interview replay#

Checkpoint#