Observability as a design component
Read firstAvailability and the nines
A design on a whiteboard always works. The real one will break at 3am in a way nobody drew, and the only question then is how fast someone can see what is wrong. That speed is decided now, in the design, by what each box reports about itself and what wakes a person up. Added after the first outage, it is always missing the one number you needed.
Logs, metrics and traces, and when to use each, are covered in logs vs metrics vs traces. This chapter is about using them as part of a design: what every service should report, how a promise like "99.9%" turns into alerts, and what to watch on the systems in this course.
Your service promises 99.9% of requests succeed over 30 days. How many minutes of complete outage does that allow in the month?
What every service reports#
Three short lists cover almost everything. They overlap on purpose.
- The four golden signals, from Google's SRE book: latency, traffic, errors and saturation. If you can only watch four things on a service, watch these.
- RED for anything that answers requests: the rate of requests, the errors, and the duration. It is the golden signals without saturation, and it is what a dashboard per service should open with.
- USE for anything that is a resource, like a disk, a queue or a connection pool: how much of it is used, how saturated it is (work waiting for it), and its errors.
Two details turn those lists into numbers you can trust.
- Latency as percentiles, never averages. An average of 80 ms can hide one request in a hundred taking four seconds. Report the 50th, 99th and 99.9th percentile, and alert on the 99th.
- Errors as a share of requests. "40 errors a minute" means nothing until you know whether that is out of 50 or 50,000.
- PrometheusMetrics: scrapes and stores time series
- GrafanaDashboards over metrics, logs and traces
- OpenTelemetryOpenTelemetry: one standard way to emit all three
- JaegerTraces: one request across services
- DatadogAll of it as a hosted product
- SentryApplication errors with their stack traces
The promise, in numbers#
An SLO (service level objective) is a target for one measurement over a window, like "99.9% of checkout requests succeed over 30 days". The gap between the target and perfect is the error budget: the amount of failure you have agreed is fine. It is the most useful number in operations, because it turns "is this bad?" into arithmetic. (SLA vs SLO vs SLI separates the three words.)
Every incident spends from the same budget. A deploy that fails 1% of requests for an hour, a slow dependency, a bad night: they all add up against 43 minutes. When the budget is nearly gone, the team slows down releases. When plenty is left, it can take more risks. That is what makes the number useful.
Pick SLOs from what users notice, measured as close to them as you can: success rate and latency of the requests they make, not CPU or memory on your machines. A user never felt a CPU graph.
Alert on what users feel#
An alert should wake someone only when users are being hurt and a person needs to act now. Everything else is a ticket for the morning, or a graph for later. Most alerting goes wrong in one of two directions: it pages on causes that hurt nobody, or it uses a fixed threshold that misses slow damage. Try three rules against the same day.
- Nightly batch job. CPU at 95% for 20 minutes. Users notice nothing.02:05 page: CPU high. Someone is woken for a batch job that hurt nobody.
- Slow leak. 0.8% of requests fail for 10 hours after a deploy. It burns 11% of the month's error budget.Missed. Nobody was told.
- Dependency outage. 20% of requests fail for 30 minutes. CPU drops to 30%, because failing requests are cheap.Missed. Nobody was told.
The third rule is burn rate alerting, from Google's SRE workbook. Burn rate is how fast the budget is being spent compared with spending it evenly over the window: 1 is exactly on budget, 10 means a 30 day budget would be gone in 3 days. Page when the rate is high over both a long and a short window, so a spike that already ended does not wake anyone. A common starting point: page above 14.4 over the last hour, which means 2% of a month's budget gone in that hour, and raise a ticket above 6 over the last six hours.
A queue holding a million messages may be fine, if it drains them in a minute. A queue holding ten may be broken, if the oldest has been there an hour. Alert on the age of the oldest message, which is what users feel as delay, not on how many are waiting.
Following one request#
A request that touches eight services fails somewhere in the middle. Logs from eight services, each with its own timestamps, will not tell you where. A trace will: every service records a span for its part of the work, tagged with one id for the whole request, and the trace view lays the spans out as a waterfall.
That only works if the id travels. The standard way is the W3C traceparent header, which
every service reads from the request it gets and passes on in every call it makes.
OpenTelemetry libraries do this for you once they are installed. The same id should appear in
every log line and in the error body you return, as the request_id from the
API design chapter, so one id ties a user's complaint to the logs and
the trace.
Tracing every request is expensive at scale, so traces are sampled. Deciding at the start of a request (keep 1 in 100) is cheap and misses rare failures. Deciding at the end, keeping every slow or failed trace and a few normal ones, catches what you need and costs a buffer to hold traces until they finish.
What to watch, design by design#
Every design in this course has a number that tells you it is healthy, and it is rarely CPU. Saying it unprompted in a round is a strong signal.
| Design | Watch this | Because |
|---|---|---|
| Notification service | Age of the oldest message per lane | Delay is what users feel, not queue size |
| CDNs and video streaming | Hit ratio at the edge | 99% to 90% is ten times the origin load |
| Ticketmaster | Share of holds that get a 409 | A rising share means fans are fighting over a stale map |
| Job scheduler | Runs started late, and lease takeovers | Late runs break the promise, takeovers mean workers are freezing |
| UPI payments | Payments stuck in DEBITED, per bank | Money that left and has not arrived |
| Real-time connections | New connections a second | A spike is a reconnect storm starting |
| Metrics system | Active series per metric | Cardinality, not volume, is what kills it |
Break it#
Read every step as text
- normal · Healthy. Services report rate, errors and duration. A handful of SLO alerts, and on call is paged a couple of times a month, each time for something real.
- alert creep · Degraded. Every incident added an alert on its cause: CPU, disk, a queue size. Now on call is paged twenty times a week, mostly for nothing, and starts acknowledging pages without reading them. The next real one is acknowledged and ignored.
- region down · Down. The region that runs the services also runs the monitoring. It goes down, and the dashboards, the alerts and the logs go down with it. Customers notice before anyone on the team does.
- repaired · Healthy. Pages only from SLO burn rate, with cause alerts turned into dashboards and tickets. Monitoring runs somewhere that does not share the services' failure, plus one simple check from outside, so silence can never look like health.
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Alert on SLO burn rate | Pages only when users are hurt fast enough to matter, and slow leaks still get noticed. | You need SLOs first, and agreement on what they are. | For anything that pages a person. |
| Percentiles instead of averages | The slow requests users actually feel are visible. | Percentiles cannot be averaged across servers, so they need histograms underneath. | Every latency number you show anyone. |
| Sampling traces at the end of the request | Every slow or failed request is kept. | Traces must be buffered until they finish, which costs memory and a collector. | Once trace volume makes keeping everything too expensive. |
| Monitoring outside the failure domain | You can see an outage that takes out a whole region. | Another region to run, and data shipped across regions. | As soon as the system runs in more than one place, and always for the one outside check. |
Interview replay#
Checkpoint#
1. 99.9% over 30 days. How many minutes of full outage does that allow?
2. A queue usually holds about 10,000 messages. Which alert is right?
3. Same service, but it now runs in two regions, and the monitoring runs in one of them. What changes first?
- 99.9% over 30 days is 43.2 minutes. That is the whole error budget.
- Four golden signals: latency, traffic, errors, saturation.
- Page on burn rate, around 14.4 over an hour. Causes like CPU go on dashboards.
- One id in the trace, every log line and the error body.
Observability is decided in the design, by what each box reports and what wakes a person. Every service reports the golden signals, latency as percentiles, traffic, errors as a share and saturation, plus one number specific to that design, like the oldest message's age in a queue or the hit ratio at a CDN. The promise is an SLO on what users feel, say 99.9 percent of checkouts over 30 days, and the gap to perfect is an error budget of 43 minutes a month that every incident spends from. Pages come only from that budget burning fast over two windows, around 14.4 over an hour, while slow leaks raise tickets and causes like CPU stay on dashboards, because high CPU often hurts nobody and many outages happen at low CPU. A trace id travels in the traceparent header through every service and appears in every log line and in the error the user sees, with slow and failed traces always kept. And monitoring runs outside the failure it watches, with one check from outside, so silence never looks like health.