Design a metrics and monitoring system
The system that tells you everything else is broken has an unusual requirement: it has to be the last thing standing. It also has an unusual workload, because the writes never stop and never slow down. Every server, every second, forever, whether anyone is looking or not.
The thing that kills these systems is not volume. It is cardinality, and it kills them suddenly, from a one line code change nobody reviewed carefully.
An engineer adds a user id label to an existing metric so they can debug one customer. What happens to your storage, and why is it worse than it sounds?
What you are building
- Ingest numeric time series from tens of thousands of hosts. Counters, gauges and histograms, at second or ten second resolution.
- Query and aggregate over time ranges. Both a dashboard covering a month and an alert covering five minutes.
- Evaluate alert rules continuously. This is the part that has to work during an incident.
- Retain data with declining resolution. A year of raw seconds is neither affordable nor useful.
- 10 million active time series.
- 20 million data points per second at peak.
- Alert evaluation within 30 seconds of the data arriving.
- The monitoring system must not depend on the systems it monitors.
- Distributed tracing and logs. Different shapes, different storage, and each is its own design question.
- Anomaly detection. It sits on top of the query layer.
- Incident management and paging workflow, which is a product rather than an infrastructure problem.
The numbers, and the one that bites
Now drag the third slider. It represents one engineer adding one label to one metric, and it multiplies everything above.
Drag the third slider from 1 to 200 and watch. That slider is one engineer adding
customer_id to a metric so they can debug one account. It is a one line change, it passes
review, and it multiplies the entire system by two hundred.
Doubling the points per series costs you disk roughly linearly, and compression absorbs a lot of it. Doubling the number of series doubles the index, doubles the open write buffers, doubles the compression state held in memory, and makes every query touch twice as many streams. Volume is a storage problem. Cardinality is a memory problem, and memory is the thing you run out of suddenly rather than gradually.
Push or pull
Storing time series
A time series is one of the most compressible things in computing, and exploiting that is what makes the whole system affordable.
Timestamps arrive at nearly fixed intervals, so store the delta of the delta, which is usually zero and costs a bit or two. Values in a series usually change slowly, so XOR each value with the previous one and store only the differing bits. Together these routinely get a 16 byte point down to under two bytes.
| series_id | bigint | PK | A hash of the metric name plus its full sorted label set. This identity is the cardinality. |
| labels | inverted index | IDX | Label to series id postings, so a query filters without scanning. This lives in memory and is what cardinality inflates. |
| chunk | compressed block | Two hours of one series, delta of delta timestamps and XOR values. Immutable once closed. | |
| head block | in memory | The current open window. Every active series holds one, which is the other memory cost of cardinality. | |
| wal | append only | Write ahead log so a crash does not lose the head block. |
Deep dive one: cardinality in practice
The lesson is that you cannot prevent this at design time, so design for containment instead. Per metric and per tenant series limits, a cardinality report that names the worst offenders, and an alert on the growth rate rather than the absolute number, because the absolute number is fine right up until it is not.
Deep dive two: retention as a pipeline
Nobody queries per second data from eight months ago, and storing it costs the same as storing today’s.
Downsample on a schedule. Raw resolution for a couple of days, one minute for a month, one hour for a year, and for each of those keep several aggregates rather than one: minimum, maximum, sum, count. Keeping only the average is the mistake, because averaging averages of different sized buckets is wrong, and the maximum is usually the number that mattered.
A query for the last thirty minutes reads raw. A query for the last quarter reads hourly rollups. The query layer picks based on the range and the pixel width of the chart, because returning three million points to draw an eight hundred pixel wide graph is work nobody benefits from.
Deep dive three: the alerting path must be independent
During an incident, the metrics system is being hammered by every engineer loading dashboards at once. That is exactly the moment alert evaluation must not slow down.
So separate them. Alert rules run on their own evaluators against their own read path, with their own capacity, and they are not competing with a hundred people running expensive ad hoc queries. A dashboard being slow during an incident is annoying. Alerts being late during an incident means nobody knows there is an incident.
The deeper version of the same rule: the monitoring system must not depend on the systems it monitors. If it authenticates through the same identity service, uses the same service discovery, and runs on the same cluster, then a failure of any of those makes the system blind precisely when it is needed. This is the question that separates people who have been on call from people who have not.
Break it
The API
{ "series": [{ "labels": {...}, "samples": [[ts, value], ...] }] }Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Pull over push | Absence is detectable, because the collector knows what should exist, and back pressure stays with the collector. | Service discovery to maintain, and short lived jobs need a gateway. | Long lived services you control. Use a push gateway as a clearly marked exception. |
| Delta of delta and XOR compression | Around ten times smaller, which changes what is affordable rather than just being an optimisation. | Points must be appended in time order, and out of order writes are expensive or rejected. | Always for regular interval numeric series. It is the reason this workload is tractable. |
| Per metric series limits | A cardinality explosion degrades one metric instead of the whole system. | Data is silently missing for the metric that hit the cap, which is confusing if the alert is missed. | Always, in any multi team system. Someone will add an unbounded label, and the only question is when. |
| Separate alert evaluation capacity | Alerts keep firing on time while everyone is loading dashboards during an incident. | More infrastructure, duplicated read capacity that is idle most of the time. | Any system where alerts page humans. This is the capacity you are paying for precisely when everything else is on fire. |
Interview replay
Checkpoint
1. Why does adding a high cardinality label hurt more than simply storing more data points?
2. What can pull based collection detect that push cannot?
3. Why keep min, max, sum and count in a downsampled rollup rather than just the average?
The workload is relentless writes and bursty reads, and the thing that actually kills these systems is cardinality rather than volume, because every distinct label combination is a separate series holding an index entry, an open write buffer and compression state, all in memory. I would collect by pull for long lived services, since that makes absence detectable and keeps back pressure with the collector, with a push gateway as a marked exception for short jobs. Storage is immutable compressed chunks per time window using delta of delta timestamps and XOR values, which is roughly ten times smaller and is what makes this affordable at all. Retention is a downsampling pipeline, raw for days then minute then hour, keeping min, max, sum and count rather than just the average. I cannot prevent a cardinality explosion at write time, so I would contain it with per metric series limits that alert naming the offending metric, and I would run alert evaluation on separate capacity, because the moment everyone is loading dashboards is exactly the moment alerts must not be late.
