35 min read

Design a metrics and monitoring system

Staff Observability Analytics Storage and indexesUpdated 28 October 2026

Read firstMessage queues Storage engines

Every app you have used has a screen somewhere showing its health: requests a second, errors, how long pages take, how full the disks are. When a number crosses a line, someone's phone rings at 3am. The system behind that screen and that phone call is a metrics and monitoring system.

It has an unusual requirement: it has to be the last thing standing, because its whole job is to tell you everything else is broken. It also has an unusual workload. The writes never stop and never slow down. Every server, every ten seconds, forever, whether anyone is looking or not. Watch one number travel from a server to a chart, and to that phone.

Targets. Service discovery tells the scrapers which services exist right now. Nobody keeps that list by hand.

A few words this page uses everywhere:

A metric is a number you measure again and again: requests a second, CPU used, queue length. Each measurement, a timestamp plus a value, is a point (also called a sample).
A time series is one metric's points in time order, like one line on a chart.
A label is a name="value" tag on a metric, such as svc="api" or code="500". Every distinct combination of labels is its own time series.
Cardinality is how many distinct series exist. It is the number that kills these systems.

A small worked example. A metric http_requests has two labels: svc with 3 values and code with 4 values. That is 3 x 4 = 12 series. Now someone adds customer_id, which has 10,000 values. The same metric is now 3 x 4 x 10,000 = 120,000 series, from a one line change. That is cardinality, and it kills these systems suddenly, from a change nobody reviewed carefully.

Metrics are one of three ways to see inside a running system, next to logs and traces. Logs vs metrics vs traces explains what each is for, and why per customer detail belongs in logs or traces rather than in a metric label.

Step 1: Understand the problem#

Six questions turn "build monitoring" into a design. Two of the answers below are the ones that matter, and neither is about how many metrics a second you can take. Find them.

You ask	They say	What it settles
How many machines, and how many metrics does each expose?	Twenty thousand hosts, about five hundred series each.	About 10 million active series, which is a storage engine decision. It is also the number that a single careless label can multiply.
At what interval, and does it ever stop?	Every ten seconds, forever, whether anyone is watching or not.	A constant write rate with no idle period. Compaction (merging small files on disk into bigger ones) has no quiet hour to run in, so background work has to survive alongside full load.
Who controls what gets labelled?	Every engineer, in their own code, with no review.	The failure mode of the whole system. Anyone can add a label, so limits and rejection have to be built in rather than left to policy.
How long do we keep it, and at what resolution?	Days at full resolution, a year rolled up.	Retention (how long data is kept) becomes a pipeline of downsampling stages rather than a delete job. Downsampling means replacing many fine points with fewer coarse ones. It also decides what a long range query is allowed to read.
Does alerting run on this same system?	Yes, and it must work while everything else is on fire.	A separate evaluation path with its own capacity. During an incident a hundred engineers are loading dashboards, and that is exactly when alerts must not queue behind them.
Is it allowed to depend on the platform it monitors?	No.	Separate cluster, separate identity, separate service discovery, and paging that leaves your infrastructure. A monitoring system that goes blind with the thing it watches is decoration.

What you are building#

Functional requirements

What the system does.

Ingest numeric time series from tens of thousands of hosts. Counters (only go up, like total requests), gauges (go up and down, like memory used) and histograms (counts per bucket, like how many requests took under 100ms), at second or ten second resolution.
Query and aggregate over time ranges. Both a dashboard covering a month and an alert covering five minutes.
Evaluate alert rules continuously. This is the part that has to work during an incident.
Retain data with declining resolution. A year of raw seconds is neither affordable nor useful.

Non-functional requirements

How well it has to do it. The numbers you commit to.

10 million active time series.
1 million data points per second: 10 million series, one point each every 10 seconds.
Alert evaluation within 30 seconds of the data arriving.
The monitoring system must not depend on the systems it monitors.

Out of scope

What you cut, and say so out loud.

Distributed tracing and logs. Different shapes, different storage, and each is its own design question.
Anomaly detection. It sits on top of the query layer.
Incident management and paging workflow, which is a product rather than an infrastructure problem.

Back of the envelope, and the number that bites#

The defaults below are the numbers from the questions above: 20,000 hosts x 500 metrics is 10,000,000 series. One point every 10 seconds is 1,000,000 points a second. At 16 bytes a point (8 for the timestamp, 8 for the value) and 8,640 points a day (86,400 seconds / 10), raw data is about 1.4 TB a day. If this kind of estimate is new, the back of the envelope chapter walks through the method.

Cardinality

Hosts20 thousand

Metrics per host500

Values on the worst label1

Active time series20k x 500 x 1 = 10,000,000

Points per second at 10s resolution1,000,000

Raw per day at 16 bytes a point1.4 TB

Compressed, roughly 10x138 GB

Index memory at ~1KB a series10.0 GB

10,000,000

series, before anyone adds an interesting label

Now drag the third slider. It represents one engineer adding one label to one metric, and it multiplies everything above.

Drag the third slider from 1 to 200 and watch. That slider is one engineer adding customer_id to a metric so they can debug one account. It is a one line change, it passes review, and it multiplies the entire system by two hundred.

Look at which rows go amber. Disk grows, but compression absorbs a lot of it. The row that fails first is index memory: the lookup table that finds a series by its labels, held in RAM so queries are fast. It needs an entry for every series, however few points that series has.

Why cardinality hurts more than volume

Doubling the points per series costs you disk roughly linearly, and compression absorbs a lot of it. Doubling the number of series doubles the index, doubles the open write buffers, doubles the compression state held in memory, and makes every query touch twice as many streams. Volume is a storage problem. Cardinality is a memory problem, and memory is the thing you run out of suddenly rather than gradually.

Step 2: Propose the high level design#

The core entities

SeriesA metric name plus one exact set of labels. Every distinct combination is its own series, held in memory while active.
PointA timestamp and a value, every 10 seconds per series. Under two bytes once compressed.
ChunkTwo hours of one series, compressed, and immutable once closed.
Alert ruleA query evaluated every 30 seconds on its own capacity, never on the dashboard path.

Never breaks: Alerts are never late because dashboards are busy.

Push or pull#

How do the numbers get from a service to the metrics system? Either the service sends them (push), or the metrics system goes and asks every service on a timer (pull, also called scraping). Think of a teacher taking attendance: calling out every name (pull) tells you who is absent, while waiting for students to sign a sheet (push) never shows the ones who did not come. Push vs pull covers the same choice for metrics with real tools. Switch between the three below.

Collection

The collector knows what should exist, so a target that stops responding is itself a signal: you can alert on a service being absent, which push can never do because a dead service simply stops sending. It also gives you natural back pressure (the receiver sets the pace, so a sender cannot flood it), since the collector controls the rate. The cost is service discovery, and difficulty reaching things behind NAT or firewalls, or short lived jobs that finish before a scrape.

The API#

Three endpoints. The first reads a range of points for a chart, the second writes points in (for the push gateway), and the third answers "which metrics have the most series right now".

GET/v1/query_range?expr=rate(http_requests{svc="api"}[5m])&start=...&end=...&step=60

returns 200 { series: [...], pointsScanned: 1840221, truncated: false }

Why: The response reports how much work it did. In a system where one careless query can scan a hundred million points, making cost visible in every response is how expensive dashboards get found before they become an outage.

POST/v1/write

{ "series": [{ "labels": {...}, "samples": [[ts, value], ...] }] }

returns 204, or 429 with the offending metric named

Why: A rejection has to say which metric hit its series limit. A generic 429 during a cardinality explosion sends an on call engineer hunting through thousands of metrics at the worst possible time.

GET/v1/cardinality?top=20

returns 200 { metrics: [{ name, series, growthPerHour }] }

Why: Growth rate matters more than the absolute number. A metric at two million series and stable is fine. One at fifty thousand and doubling every ten minutes is the deploy that is about to take the system down.

The whole system on one whiteboard#

Figure 1. Two paths leave the time series database and they are deliberately not the same path. Dashboards can queue behind each other during an incident. Alert evaluation cannot.

Walking Figure 1:

Scrapers pull from targets on a fixed interval. Service discovery (a live list of which services and machines exist right now) tells them what to scrape, which is the difference between monitoring a fleet and monitoring a list somebody maintained by hand. There are several scrapers, each owning a share of the targets.
Every sample passes a cardinality check on the way in. A new label combination is a new series, and a new series is a permanent cost, so this is where you refuse rather than in a postmortem.
Accepted samples are appended to the shard that owns that series. A shard is one machine's slice of the data; the series id decides which machine (the sharding chapter covers how).
A downsampler produces coarser copies in the background. Nothing is deleted by a query, only by retention.
The query layer chooses which resolution answers a given range.
Dashboards are served from there and are allowed to be slow.
Alert evaluation reads its own path with its own capacity. This is the arrow people forget, and the one that matters at 3am.
Firing alerts leave through a notifier that does not live on this cluster. (Sending the page itself is a small version of design a notification service.)
Which is how the page gets out even when the platform you are monitoring is the thing that is down.

Storing time series#

A time series is one of the most compressible things in computing, and that is what makes the whole system affordable. Two tricks do most of the work.

Timestamps. Points arrive every 10 seconds, so the gap between two timestamps is nearly always 10. Store the change in the gap instead, the delta of delta: 10 minus 10 is 0, and a zero costs a single bit. Only a late scrape costs more.

Values. A value usually changes slowly: 12, 12, 13, 13. Compare each value's 64 bits with the previous value's using XOR (a bit by bit "are these different?") and keep only the bits that changed. An identical value costs a single bit.

Together these routinely get a 16 byte point down to under two bytes. Try both kinds of data below.

Squeezing eight points

Time	Gap	Change in gap	Value	Bits changed	Cost
1000			12		128 bits, stored whole
1010	10	0	12	0	2 bits
1020	10	0	12	0	2 bits
1030	10	0	13	1	15 bits
1040	10	0	13	0	2 bits
1051	11	1	13	0	10 bits
1060	9	-2	13	0	10 bits
1070	10	1	12	1	23 bits

128 bytes raw become 24 bytes. After the first point, which is stored whole, each point costs about 1.1 bytes instead of 16. Most gaps are exactly 10 seconds and most values repeat, so most fields cost a single bit.

Each point starts as 16 bytes: an 8 byte timestamp and an 8 byte value. Costs here use one simple rule: a zero costs 1 bit, a small change in the gap costs 9 bits, and a changed value costs its changed bits plus 13 bits saying where they are. The scrape interval is 10 seconds, and one scrape arrived a second late.

The steady series shrinks to about a byte a point. The noisy one only halves, because almost every bit of each value changes. Real metrics look much more like the first one, which is why the trick works.

The data is kept in immutable chunks: files covering one window of time that are written once and never edited. It is the same idea as an LSM tree, which only ever appends. Recent points sit in an open in memory block, the head block, protected by a write ahead log (every write is appended to a file first, so a crash can be replayed). A chunk keeps one series' points next to each other on disk, so drawing one line on a chart reads one block instead of picking values out of many. Analytics databases use the same idea when they store each column together; OLTP vs OLAP shows why that layout makes this kind of read fast.

series and chunkscolumn oriented, immutable chunks per time window

series_id	bigint	PK	A hash (a short fingerprint) of the metric name plus its full sorted label set. One id per distinct combination, so counting these ids is counting the cardinality.
labels	inverted index	IDX	For each label value, the list of series ids that have it, like the index at the back of a book. A query filters without scanning. This lives in memory and is what cardinality inflates.
chunk	compressed block		Two hours of one series, delta of delta timestamps and XOR values. Immutable once closed.
head block	in memory		The current open window. Every active series holds one, which is the other memory cost of cardinality.
wal	append only		Write ahead log so a crash does not lose the head block.

Sample row

series 8814 = http_requests{svc="api",code="500"} | chunk 14:00-16:00 | 720 points in about 1.2KB

Note where memory actually goes: the label index and one open head block per active series. Both scale with the number of series and neither scales with how long you retain data, which is exactly why cardinality and volume fail in completely different ways.

Your answer

An engineer adds a user id label to an existing metric so they can debug one customer. Which box in Figure 1 is supposed to stop them, and what happens to storage if it does not?

Step 3: Design deep dive#

Cardinality in practice#

Below, each bar is one metric, and its height is that metric's share of all the series in the system. Switch to the second option to see one bad deploy.

Where the series are

Metric 0 carries 14.0% (even would be 12.5%)

Series spread across metrics roughly evenly, because every metric is labelled by things with small bounded value sets: service, instance, status code, region. This is what you designed for.

You cannot prevent this at design time, so design for containment instead.

How do you stop one label from taking the whole system down?

First try
Ask teams not to
Write a guideline: never put user ids, request ids or emails in labels. Review it in onboarding.
Where it breaks: Every engineer adds labels in their own code with no review. One person debugging one customer at 2am adds customer_id, and the guideline is not in the room.
Better
One global series limit
Cap the total number of series the system accepts. Past the cap, new series are rejected, so memory can never run out.
Where it breaks: The bad metric eats the whole budget. Every other team's new series are now rejected too, so one deploy degrades everyone, and nobody can tell which metric caused it.
What you ship
Limits per metric and per team, an alert on growth, and a report
Each metric and each team (tenant) gets its own series limit. Past it, new label combinations are rejected and an alert names the metric. A cardinality report ranks metrics by how fast they are growing, because a metric doubling every ten minutes is the next outage even while its total still looks small.

The offending metric is degraded, and everything else, including the ability to see that it is degraded, keeps working. That is the right blast radius.

Retention as a pipeline#

Nobody queries per second data from eight months ago, and storing it costs the same as storing today's.

How do you keep a year of data without paying for a year of raw points?

First try
Keep every raw point
Store everything at 10 second resolution for a year.
Where it breaks: Storage grows by the same amount every day, forever. And a chart of the last quarter has to read and draw about 778,000 points per series (90 days x 8,640) to fill a chart a few hundred pixels wide.
Better
Roll up to averages
After a few days, replace each minute of points with one average, and later each hour with one average.
Where it breaks: A one minute spike to 3 seconds of latency vanishes into an hourly average of 200ms. And averaging averages of buckets with different numbers of points gives the wrong answer.
What you ship
Downsample in stages, keeping min, max, sum and count
Raw for a couple of days, one minute for a month, one hour for a year. Each rollup keeps the minimum, maximum, sum and count. Sum divided by count gives a correct average over any range, and the maximum keeps the spike. The query layer picks the resolution from the range asked for.

Figure 2. Data moves rightwards over time, getting coarser and cheaper. Queries pick the resolution from the range asked for.

A query for the last thirty minutes reads raw. A query for the last quarter reads hourly rollups. The query layer picks based on the range and the pixel width of the chart, because returning three million points to draw an eight hundred pixel wide graph is work nobody benefits from.

Why min, max, sum and count, worked out. Two rollup buckets: one holds 2 points averaging 100ms, the other 8 points averaging 10ms. The average of the two averages is 55ms. The real average is (2 x 100 + 8 x 10) / 10 = 28ms. With sum and count kept, you get 28ms every time.

The alerting path must be independent#

During an incident, the metrics system is being hammered by every engineer loading dashboards at once. That is exactly the moment alert evaluation must not slow down.

Where do alert rules run?

First try
As ordinary queries on the shared query layer
An alert rule is just a query on a timer, so run it through the same query layer the dashboards use.
Where it breaks: At 3am a hundred engineers open dashboards and run expensive ad hoc queries. Alert rules queue behind them, and the alerts for the incident fire minutes late.
Better
Separate evaluators with their own read path
Alert evaluators get their own capacity and read the data directly, so dashboard load cannot delay them.
Where it breaks: They still log in through the company identity service, find targets through the shared service discovery and run on the shared cluster. When one of those fails, the system that should report the failure goes blind with it.
What you ship
Own capacity, and independent of what it watches
Separate evaluators and read path, plus a separate cluster, separate identity, separate service discovery, and a notifier whose pages leave your infrastructure. The monitoring system keeps working when the platform it monitors does not.

A dashboard being slow during an incident is annoying. Alerts being late during an incident means nobody knows there is an incident. This is the question that separates people who have been on call from people who have not.

The independence rule is the weakest link rule turned around: every shared dependency is a link in the monitoring system's chain. Which rules to page on is its own topic; SLA vs SLO vs SLI explains alerting on burn rate, which ties pages to what users actually feel.

Break it#

Step through what goes wrong, in order. The last step is the repaired design.

Ingest health

normal

normalcardinality spikeingesters OOMincident loadcontained

Healthy. 10 million series, 1 million points a second, alert rules evaluating every 30 seconds and finishing on time. Compression is doing most of the work and nobody is thinking about the monitoring system.

Read every step as text

normal · Healthy. 10 million series, 1 million points a second, alert rules evaluating every 30 seconds and finishing on time. Compression is doing most of the work and nobody is thinking about the monitoring system.
cardinality spike · Degraded. A deploy adds a request id label to one metric. Series count climbs from 10 million towards 300 million over the next few minutes. Every new series allocates an index entry and an open head block, so ingester memory climbs far faster than disk usage does.
ingesters OOM · Down. Ingesters (the machines that accept writes) run out of memory (OOM) and restart, losing head blocks and replaying write ahead logs, which is slow and memory hungry, so they run out again. The monitoring system is now down, which means nobody can see that anything else is wrong either. This is the worst outage in this design because it hides itself.
incident load · Down. A separate scenario with the same ending. A real incident starts, a hundred engineers open dashboards, and expensive queries saturate the read path. Alert evaluation shares that path and starts running late, so alerts for the actual incident fire minutes after they should.
contained · Degraded. Per metric series limits reject new label combinations past a threshold and alert loudly naming the metric, so one bad deploy degrades one metric instead of the system. Alert evaluators run on separate capacity with their own read path, so dashboard load cannot delay them. The offending metric is broken and everything else, including the ability to see that it is broken, still works.

Trade-offs#

Choice	What you gain	What you pay	Pick it when
Pull over push	Absence is detectable, because the collector knows what should exist, and back pressure stays with the collector.	Service discovery to maintain, and short lived jobs need a gateway.	Long lived services you control. Use a push gateway as a clearly marked exception.
Delta of delta and XOR compression	Around ten times smaller, which changes what is affordable rather than just being an optimisation.	Points must be appended in time order, and out of order writes are expensive or rejected.	Always for regular interval numeric series. It is the reason this workload is tractable.
Per metric series limits	A cardinality explosion degrades one metric instead of the whole system.	Data is silently missing for the metric that hit the cap, which is confusing if the alert is missed.	Always, in any multi team system. Someone will add an unbounded label, and the only question is when.
Separate alert evaluation capacity	Alerts keep firing on time while everyone is loading dashboards during an incident.	More infrastructure, duplicated read capacity that is idle most of the time.	Any system where alerts page humans. This is the capacity you are paying for precisely when everything else is on fire.

The level bar#

One moment from this round, answered three ways. Pick the one you think is staff level before the labels appear. (What changes at senior and staff explains the gap.)

Pick the staff level answer

InterviewerA team wants to label every request metric with customer id, for about 50,000 customers. Do you allow it?

Show which is which without guessing

Answer A: Staff. Turns a no into the cheapest thing that answers the real question, and keeps the guard that makes mistakes survivable.
Answer B: Senior. Correct, with the right alternative. It refuses without solving what the team actually needs.
Answer C: Mid level. Every distinct label combination is a separate series in memory, so this multiplies the series count by 50,000.

Then run the question cold, with a 45 minute timer and no notes, and score the run on the five things this question is really testing.

Score your last run

0 / 10

What kills it
Collection
Storage
Retention
During an incident

Pick 0, 1 or 2 on every row. Be the strict interviewer, not the kind friend.

Interview replay#

Interviewer

What is the hardest part of running this at scale?

Open. The answer they are listening for is a specific word.

You

Cardinality. Volume is a storage problem and compression handles most of it, but every distinct label combination is a separate series with its own index entry, its own open write buffer and its own compression state, and all of that is memory. One engineer adding a user id label to an existing metric can multiply the series count by a thousand in a single deploy, and memory is the thing you run out of suddenly rather than gradually.

Names it in the first word, then explains the mechanism rather than just the symptom.

Interviewer

Can you prevent it?

The interesting follow up, because the honest answer is no.

You

Not at write time, because there is no way to tell an unbounded label from a legitimate one when the first few values arrive. So I would contain it instead: a per metric series limit that rejects new label combinations past a threshold and alerts naming the metric, plus a cardinality report ranked by growth rate rather than absolute count. One bad metric then degrades itself instead of the system.

Saying prevention is impossible and moving to containment is the senior move. Growth rate over absolute count is the detail that shows operational experience.

Interviewer

Push or pull for collection?

A preference question with one strong argument on the pull side.

You

Pull for anything long lived, mainly because absence becomes observable: the collector knows what should be there, so a target that stops responding is a signal, whereas with push a dead service looks identical to one that was never configured. Pull also keeps back pressure on the collector rather than letting a service in a retry loop flood ingest. I would add a push gateway for short lived batch jobs, and keep it clearly an exception rather than a second default.

The absence argument is the one that decides it, and it is stated first.

Interviewer

It is 3am, there is a major incident, and everyone is loading dashboards. What breaks?

The question that separates people who have carried a pager.

You

Alert evaluation, if it shares a read path with dashboards. A hundred expensive ad hoc queries saturate the query layer and alert rules start evaluating late, so alerts for the incident fire minutes after they should. I would give alert evaluators separate capacity and their own read path. More generally the monitoring system must not depend on what it monitors: same cluster, same identity service, same service discovery, and it goes blind exactly when you need it.

Answers the immediate case and then generalises to the independence principle. The generalisation is what makes it a staff level answer.

Interviewer

How long would you keep raw resolution data?

Checking whether retention is thought of as a pipeline or a setting.

You

A couple of days, then downsample to one minute for a month and one hour for a year. The important detail is keeping minimum, maximum, sum and count in each rollup rather than just the average, because averaging averages across unequal buckets is wrong and the maximum is usually the number someone actually cared about. I would set the exact windows from what people query, which I would measure rather than guess, since query ranges cluster far more tightly than anyone expects.

The min max sum count detail is small, specific and very hard to fake.

Checkpoint#

Checkpoint

1. Why does adding a high cardinality label hurt more than simply storing more data points?

2. What can pull based collection detect that push cannot?

3. Why keep min, max, sum and count in a downsampled rollup rather than just the average?

Worth memorising

20,000 hosts x 500 series is 10 million active series, and 1 million points a second at a 10 second interval.
One careless label multiplies the series count, and series count is what kills these systems, not volume.
Every 10 seconds, forever. There is no quiet hour to run compaction in.
Under two bytes a point after delta of delta timestamps and XOR values, down from 16.
Alerting reads its own path. During an incident dashboards may be slow and alerts may not.

Say this in 60 seconds

The workload is relentless writes and bursty reads, and the thing that actually kills these systems is cardinality rather than volume, because every distinct label combination is a separate series holding an index entry, an open write buffer and compression state, all in memory. I would collect by pull for long lived services, since that makes absence detectable and keeps back pressure with the collector, with a push gateway as a marked exception for short jobs. Storage is immutable compressed chunks per time window using delta of delta timestamps and XOR values, which is roughly ten times smaller and is what makes this affordable at all. Retention is a downsampling pipeline, raw for days then minute then hour, keeping min, max, sum and count rather than just the average. I cannot prevent a cardinality explosion at write time, so I would contain it with per metric series limits that alert naming the offending metric, and I would run alert evaluation on separate capacity, because the moment everyone is loading dashboards is exactly the moment alerts must not be late.