Mitigation vs root cause
Published
On 2 July 2019, Cloudflare pushed a new firewall rule containing a regular expression that backtracked badly. CPUs on its machines around the world went to 100% and a large share of the sites behind Cloudflare started returning errors. Engineers did not fix the regular expression first. They used a switch built for exactly this, turned off that whole set of firewall rules everywhere at once, and traffic recovered. Working out exactly what was wrong with the regular expression came later, after the sites were back.
That is the order every incident should run in. Mitigation is whatever stops the damage now: roll back, switch a feature off, move traffic, add capacity. Root cause is the full understanding of why it happened. The rule is simple and hard to follow at 3am: mitigate first, investigate second. You do not need to understand a fire to close the gas valve.
You are paged at night: 14% of checkout requests are failing. A deploy went out 12 minutes ago. You think you might see the bug in the diff. What do you do first, and why?
Being on call#
On call means that for a week or so, you are the person who gets paged when a service you own breaks, day or night. Most teams rotate it so everyone takes a turn. Teams spread across time zones can do better: follow the sun, where India covers its daytime, then Europe, then the US, and nobody gets paged at 3am. That only works if each region hands over cleanly, which is the same skill as any async handoff.
Your on-call week is only as good as your alerts. The rule from Google's SRE books is that a page should mean a human needs to act now, on something users can feel. Everything else is a ticket for working hours, or a dashboard.
An alert that fires and needs no action does real harm. After the tenth false alarm the on-call person stops taking the eleventh seriously. Delete noisy alerts, or turn them into tickets.
Running the incident#
Here is a page from a real kind of incident. Play it through, one choice at a time.
Your phone goes off. "checkout-api: error budget burning 30x. 14% of checkout requests failing for 10 minutes."
The best path was the same at every step: stop the damage, then understand it. A rollback you can do without understanding the bug beats a fix you understand but have to write, test and deploy at 3am.
The levers for mitigation are always roughly the same, so know where they are before you need them:
- Roll back the last deploy. Most incidents follow a change, so this fixes most of them.
- Turn off a feature flag, if the new code is behind one. Faster than a rollback.
- Move traffic away: to another region, or off a bad machine or zone.
- Shed load: rate limit or turn off non-essential features so the important path survives.
- Add capacity, if the problem really is load.
For anything bigger than one person can handle, split the work. Google's SRE book borrowed this from the incident command system US firefighters use: an incident commander who coordinates and decides, an operations lead who changes things, and a communications lead who posts updates. The commander's job is not to debug. It is to make sure someone is, and that everyone else knows what is happening.
On a remote team nobody can see you working. Post an update every 15 to 30 minutes, even if it is "still investigating, rollback in progress, next update at 03:30". Without it, people start messaging you for news, which slows the person who is fixing it.
The postmortem#
Once things are stable, the team writes a postmortem: what happened, why, and what changes so it can't happen the same way again. The important word is blameless. A postmortem asks what about the system let a person make this mistake, and how to close that gap. It never asks who to blame.
That isn't about being kind for the sake of it. If people get blamed for incidents, they hide mistakes and report problems late, and late reports make incidents longer. John Allspaw wrote about this at Etsy in 2012, and Google's SRE book gives postmortem culture a chapter of its own.
A good postmortem has:
- A summary: what users saw, for how long, and how many were affected.
- A timeline, with times: when it started, when it was noticed, when it was mitigated. The gaps between those times are usually where the lessons are.
- Contributing factors, plural. Real incidents have several: the bug, the missing test, the skipped canary, the slow alert. "Root cause" in the singular often hides the others.
- Action items, each with an owner and a date. "Be more careful" is not an action item.
On 28 February 2017 an engineer debugging the Amazon S3 billing system in us-east-1 mistyped an input to a command and removed far more servers than intended. S3 in that region was down for about four hours. Amazon's public write-up does not blame the engineer. It changes the tool, so it removes capacity more slowly and refuses to go below a safe minimum. On 4 October 2021 a maintenance command took down Facebook's backbone network for about six hours, and the outage also cut off the internal tools engineers needed to fix it. Both are short, readable, and worth an evening.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Mitigate first | Users stop being hurt within minutes, often before anyone knows what the bug is. | The evidence can disappear with the rollback, so grab logs, graphs and the bad version number first. | Always, during an incident. Understanding comes after users are safe. |
| Root cause first | You fix the real problem in one go, if you find it fast. | Users keep failing while you read code at 3am, and a rushed fix can start a second incident. | Only when no mitigation exists, for example data corruption that a rollback would not undo. |
| Blameless postmortem | People report problems early, and fixes target the system, so the same incident does not repeat. | Takes a few hours to write well, and needs leaders who mean "blameless". | After every incident users noticed, and after near misses worth learning from. |
- PaPagerDutyPaging, on-call schedules, escalation. Opsgenie is the other big one
- DatadogMonitoring and alerting
- GrafanaDashboards, often on top of Prometheus
- SentryError tracking, grouped by release
In the interview#
Senior and remote interviews both ask about incidents, as a behavioural question and inside design questions ("what happens when this breaks?").
Checkpoint#
1. Errors started right after a deploy. You think you can see the bug in the diff. What first?
2. Which should page someone at night?
3. A postmortem says "Root cause: human error. Action: be more careful." What is wrong?
- Mitigate first, investigate second. Rollback, flag off, move traffic, shed load.
- A page means a human must act now on something users feel. Everything else is a ticket.
- Update every 15 to 30 minutes during an incident, even if nothing changed.
- Postmortems are blameless: timeline, contributing factors, actions with owners and dates.
- Capture evidence before you roll back: the version, the graphs, a sample of the errors.
During an incident, mitigation comes before root cause. Mitigation is anything that stops users being hurt now: rolling back, turning off a flag, moving traffic, shedding load. Root cause is understanding why, and it can wait until users are safe. A page should only fire when a human must act now on something users feel, like a fast error budget burn, and everything else becomes a ticket, because noisy alerts train people to ignore real ones. For bigger incidents we split roles: an incident commander who coordinates, someone who operates, someone who communicates, with updates every 15 to 30 minutes. Afterwards we write a blameless postmortem with a timeline, several contributing factors, and action items with owners, so the system changes and not just the person.