Blue-green vs canary vs rolling
Published
Every deploy is a bet that the new version works. Deployment strategies decide how much you bet at once and how fast you can take it back.
- Rolling: replace instances a few at a time until all run the new version. Like replacing the tyres of a bus one wheel at a time, while it keeps driving.
- Blue-green: build a complete second environment (green) with the new version next to the live one (blue), then switch all traffic over at once. Keep blue around so you can switch back.
- Canary: send a small slice of traffic, 1% or 5%, to the new version, watch its metrics, and widen step by step only while it looks healthy. Named after the canaries miners carried to detect gas before it reached them.
A release has a bug that fails every checkout it handles. Your monitoring takes about 6 minutes to fire. With each of the three strategies, roughly what share of users would hit the bug before you recover?
Play it out#
Version 2 fails every request it serves. Move the slider for how quickly you notice.
Three things stand out. Blue-green bets everything at once, so the exposure is everyone, for as long as it takes to notice, but the rollback is a single switch. Rolling spreads the bet out, but keeps spreading it until someone notices, and rolling back is another slow rollout. Canary keeps the bet small until the evidence is in, so even a slow alert hurts few users, provided the canary's metrics are watched separately from the rest.
How each works in practice#
Rolling is the default in Kubernetes Deployments: maxSurge and maxUnavailable decide
how many pods are added and removed per step, and readiness probes stop the rollout if new
pods do not become healthy. Cheap (you need at most a few extra instances) and automatic, but
for a while two versions serve traffic together, and the rollback is another rollout.
Blue-green needs a second full environment and a switch: a load balancer target group, a DNS record, or a Kubernetes Service selector. You can test green with real infrastructure before any user sees it, and roll back in seconds. It costs double capacity during the switch, and the database is shared by both colours, which is the hard part (below).
Canary needs traffic splitting by percentage (a service mesh, an L7 load balancer, or a tool like Argo Rollouts or Flagger) and, above all, automated analysis: compare the canary's error rate and latency with the baseline and roll back automatically if they diverge. Netflix's Kayenta popularised automated canary analysis. Without metrics split by version, a 5% canary is just a slow rolling deploy.
The database breaks all three#
Every strategy has old and new code running against the same database at some point, either side by side (rolling, canary) or one after the other with a rollback (blue-green). So a schema change must work with both versions. The pattern is expand and contract:
- Expand: add the new column or table, nullable, without removing anything. Old code ignores it.
- Deploy code that writes to both old and new, and reads the old. Backfill existing rows.
- Deploy code that reads the new.
- Contract: once no running version uses the old column, drop it, in a later release.
Renaming a column in one step is the classic way to make a "safe" canary take down production: the old version's queries fail the moment the migration runs.
On 19 July 2024 a faulty CrowdStrike content update was pushed to Windows machines worldwide in a short window, and about 8.5 million devices crashed into boot loops: airlines, hospitals, banks and broadcasters went down together. CrowdStrike's own review committed to staged deployment of these updates, starting with canary groups. A rollout strategy is not paperwork. It decides the size of your worst day.
Deploy is not release#
Feature flags separate shipping code from turning it on. The code goes out dark with any strategy, then a flag enables it for employees, then 1% of users, then everyone, and can be turned off in seconds without a deploy. Canary releases by flag and canary deploys by traffic split solve the same problem at different layers, and teams that move fast use both.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Rolling | Built into Kubernetes, little extra capacity, automatic. | Bad versions keep spreading until noticed; rollback is slow; two versions live together. | Low risk services, internal tools, the default when nothing else is set up. |
| Blue-green | Test the full new environment before users see it; instant switch and instant rollback. | Double capacity during the switch; every user is exposed at once; shared database must suit both. | Releases you want to verify end to end first, stateful apps hard to run side by side, quick rollback needs. |
| Canary | Smallest blast radius, real production evidence, automatic rollback. | Needs traffic splitting and metrics per version; slower full rollout. | High traffic, high risk services: payments, checkout, anything customer facing at scale. |
- KubernetesRolling updates by default (maxSurge, maxUnavailable)
- ArgoArgo Rollouts: canary and blue-green with analysis
- IstioPercentage traffic splitting between versions
- AWSCodeDeploy blue-green, ALB weighted target groups
In the interview#
Checkpoint#
1. Which strategy exposes the fewest users to a bad release?
2. What does a canary need to actually reduce risk?
3. Why is renaming a column in one migration dangerous during a rolling deploy?
- Rolling: a few instances at a time. Blue-green: full switch with a way back. Canary: small slice, then widen.
- Exposure vs rollback speed is the trade.
- Canary only works with per version metrics and automatic rollback.
- Schema changes: expand and contract across releases.
- Feature flags separate deploy from release.
Deployment strategies decide how much risk you take at once and how fast you can undo it. Rolling updates replace instances a few at a time, which Kubernetes does by default; it is cheap, but a bad version keeps spreading until noticed and rolling back is another rollout. Blue-green builds a full second environment and switches all traffic at once; you can verify it first and roll back instantly, at the cost of double capacity and exposing everyone at the switch. Canary sends a small slice of traffic to the new version and widens in steps while automated analysis compares its errors, latency and business metrics with the baseline, rolling back automatically; it has the smallest blast radius but needs traffic splitting and per version metrics. In every strategy two versions share the database, so schema changes use expand and contract, and feature flags let me separate deploying code from releasing behaviour.