Design a metrics and monitoring system
The system that tells you everything else is broken has an unusual requirement: it has to be the last thing standing. It also has an unusual workload, because the writes never stop and never slow down. Every server, every second, forever, whether anyone is looking or not.
The thing that kills these systems is not volume. It is cardinality, and it kills them suddenly, from a one line code change nobody reviewed carefully.
Step 1: Understand the problem#
Two of the answers below are the ones that matter, and neither is about how many metrics a second you can take. Find them.
| You ask | They say | What it settles |
|---|---|---|
| How many machines, and how many metrics does each expose? | A hundred thousand hosts, a few hundred series each. | Tens of millions of active series, which is a storage engine decision. It is also the number that a single careless label can multiply. |
| At what interval, and does it ever stop? | Every ten seconds, forever, whether anyone is watching or not. | A constant write rate with no idle period. There is no quiet hour to run compaction in, so background work has to survive alongside full load. |
| Who controls what gets labelled? | Every engineer, in their own code, with no review. | The failure mode of the whole system. Cardinality is a self service footgun, so limits and rejection have to be built in rather than left to policy. |
| How long do we keep it, and at what resolution? | Days at full resolution, a year rolled up. | Retention as a pipeline of downsampling stages rather than a delete job. It also decides what a long range query is allowed to read. |
| Does alerting run on this same system? | Yes, and it must work while everything else is on fire. | A separate evaluation path with its own capacity. During an incident a hundred engineers are loading dashboards, and that is exactly when alerts must not queue behind them. |
| Is it allowed to depend on the platform it monitors? | No. | Separate cluster, separate identity, separate service discovery, and paging that leaves your infrastructure. A monitoring system that goes blind with the thing it watches is decoration. |
What you are building#
- Ingest numeric time series from tens of thousands of hosts. Counters, gauges and histograms, at second or ten second resolution.
- Query and aggregate over time ranges. Both a dashboard covering a month and an alert covering five minutes.
- Evaluate alert rules continuously. This is the part that has to work during an incident.
- Retain data with declining resolution. A year of raw seconds is neither affordable nor useful.
- 10 million active time series.
- 20 million data points per second at peak.
- Alert evaluation within 30 seconds of the data arriving.
- The monitoring system must not depend on the systems it monitors.
- Distributed tracing and logs. Different shapes, different storage, and each is its own design question.
- Anomaly detection. It sits on top of the query layer.
- Incident management and paging workflow, which is a product rather than an infrastructure problem.
Back of the envelope, and the number that bites#
Now drag the third slider. It represents one engineer adding one label to one metric, and it multiplies everything above.
Drag the third slider from 1 to 200 and watch. That slider is one engineer adding
customer_id to a metric so they can debug one account. It is a one line change, it passes
review, and it multiplies the entire system by two hundred.
Doubling the points per series costs you disk roughly linearly, and compression absorbs a lot of it. Doubling the number of series doubles the index, doubles the open write buffers, doubles the compression state held in memory, and makes every query touch twice as many streams. Volume is a storage problem. Cardinality is a memory problem, and memory is the thing you run out of suddenly rather than gradually.
Step 2: Propose the high level design#
Push or pull#
The API#
{ "series": [{ "labels": {...}, "samples": [[ts, value], ...] }] }The whole system on one whiteboard#
Walking Figure 1:
- Scrapers pull from targets on a fixed interval. Service discovery tells them what exists, which is the difference between monitoring a fleet and monitoring a list somebody maintained by hand.
- Every sample passes a cardinality check on the way in. A new label combination is a new series, and a new series is a permanent cost, so this is where you refuse rather than in a postmortem.
- Accepted samples are appended to the shard that owns that series.
- A downsampler produces coarser copies in the background. Nothing is deleted by a query, only by retention.
- The query layer chooses which resolution answers a given range.
- Dashboards are served from there and are allowed to be slow.
- Alert evaluation reads its own path with its own capacity. This is the arrow people forget, and the one that matters at 3am.
- Firing alerts leave through a notifier that does not live on this cluster.
- Which is how the page gets out even when the platform you are monitoring is the thing that is down.
Storing time series#
A time series is one of the most compressible things in computing, and exploiting that is what makes the whole system affordable.
Timestamps arrive at nearly fixed intervals, so store the delta of the delta, which is usually zero and costs a bit or two. Values in a series usually change slowly, so XOR each value with the previous one and store only the differing bits. Together these routinely get a 16 byte point down to under two bytes.
| series_id | bigint | PK | A hash of the metric name plus its full sorted label set. This identity is the cardinality. |
| labels | inverted index | IDX | Label to series id postings, so a query filters without scanning. This lives in memory and is what cardinality inflates. |
| chunk | compressed block | Two hours of one series, delta of delta timestamps and XOR values. Immutable once closed. | |
| head block | in memory | The current open window. Every active series holds one, which is the other memory cost of cardinality. | |
| wal | append only | Write ahead log so a crash does not lose the head block. |
An engineer adds a user id label to an existing metric so they can debug one customer. Which box in Figure 1 is supposed to stop them, and what happens to storage if it does not?
Step 3: Design deep dive#
Cardinality in practice#
The lesson is that you cannot prevent this at design time, so design for containment instead. Per metric and per tenant series limits. A cardinality report that names the worst offenders by team. And an alert on the growth rate rather than the absolute number, because the absolute number looks fine right up until it does not.
Retention as a pipeline#
Nobody queries per second data from eight months ago, and storing it costs the same as storing today's.
Downsample on a schedule. Raw resolution for a couple of days, one minute for a month, one hour for a year, and for each of those keep several aggregates rather than one: minimum, maximum, sum, count. Keeping only the average is the mistake, because averaging averages of different sized buckets is wrong, and the maximum is usually the number that mattered.
A query for the last thirty minutes reads raw. A query for the last quarter reads hourly rollups. The query layer picks based on the range and the pixel width of the chart, because returning three million points to draw an eight hundred pixel wide graph is work nobody benefits from.
The alerting path must be independent#
During an incident, the metrics system is being hammered by every engineer loading dashboards at once. That is exactly the moment alert evaluation must not slow down.
So separate them. Alert rules run on their own evaluators against their own read path, with their own capacity, and they are not competing with a hundred people running expensive ad hoc queries. A dashboard being slow during an incident is annoying. Alerts being late during an incident means nobody knows there is an incident.
The deeper version of the same rule: the monitoring system must not depend on the systems it monitors. If it authenticates through the same identity service, uses the same service discovery, and runs on the same cluster, then a failure of any of those makes the system blind precisely when it is needed. This is the question that separates people who have been on call from people who have not.
Break it#
Read every step as text
- normal · Healthy. 10 million series, 20 million points a second, alerts evaluating every 15 seconds well inside their window. Compression is doing most of the work and nobody is thinking about the monitoring system.
- cardinality spike · Degraded. A deploy adds a request id label to one metric. Series count climbs from 10 million towards 300 million over the next few minutes. Every new series allocates an index entry and an open head block, so ingester memory climbs far faster than disk usage does.
- ingesters OOM · Down. Ingesters run out of memory and restart, losing head blocks and replaying write ahead logs, which is slow and memory hungry, so they run out again. The monitoring system is now down, which means nobody can see that anything else is wrong either. This is the worst outage in this design because it hides itself.
- incident load · Down. A separate scenario with the same ending. A real incident starts, a hundred engineers open dashboards, and expensive queries saturate the read path. Alert evaluation shares that path and starts running late, so alerts for the actual incident fire minutes after they should.
- contained · Degraded. Per metric series limits reject new label combinations past a threshold and alert loudly naming the metric, so one bad deploy degrades one metric instead of the system. Alert evaluators run on separate capacity with their own read path, so dashboard load cannot delay them. The offending metric is broken and everything else, including the ability to see that it is broken, still works.
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Pull over push | Absence is detectable, because the collector knows what should exist, and back pressure stays with the collector. | Service discovery to maintain, and short lived jobs need a gateway. | Long lived services you control. Use a push gateway as a clearly marked exception. |
| Delta of delta and XOR compression | Around ten times smaller, which changes what is affordable rather than just being an optimisation. | Points must be appended in time order, and out of order writes are expensive or rejected. | Always for regular interval numeric series. It is the reason this workload is tractable. |
| Per metric series limits | A cardinality explosion degrades one metric instead of the whole system. | Data is silently missing for the metric that hit the cap, which is confusing if the alert is missed. | Always, in any multi team system. Someone will add an unbounded label, and the only question is when. |
| Separate alert evaluation capacity | Alerts keep firing on time while everyone is loading dashboards during an incident. | More infrastructure, duplicated read capacity that is idle most of the time. | Any system where alerts page humans. This is the capacity you are paying for precisely when everything else is on fire. |
Interview replay#
Checkpoint#
1. Why does adding a high cardinality label hurt more than simply storing more data points?
2. What can pull based collection detect that push cannot?
3. Why keep min, max, sum and count in a downsampled rollup rather than just the average?
- 100,000 hosts with a few hundred series each is tens of millions of active series.
- One careless label multiplies the series count, and series count is what kills these systems, not volume.
- Every 10 seconds, forever. There is no quiet hour to run compaction in.
- Alerting reads its own path. During an incident dashboards may be slow and alerts may not.
The workload is relentless writes and bursty reads, and the thing that actually kills these systems is cardinality rather than volume, because every distinct label combination is a separate series holding an index entry, an open write buffer and compression state, all in memory. I would collect by pull for long lived services, since that makes absence detectable and keeps back pressure with the collector, with a push gateway as a marked exception for short jobs. Storage is immutable compressed chunks per time window using delta of delta timestamps and XOR values, which is roughly ten times smaller and is what makes this affordable at all. Retention is a downsampling pipeline, raw for days then minute then hour, keeping min, max, sum and count rather than just the average. I cannot prevent a cardinality explosion at write time, so I would contain it with per metric series limits that alert naming the offending metric, and I would run alert evaluation on separate capacity, because the moment everyone is loading dashboards is exactly the moment alerts must not be late.