Library
9 min read

Feature flags vs long-lived branches

Published

You are building a new checkout that will take three weeks. Users must not see it until it is done. There are two ways to hide it.

A long-lived branch keeps the new code out of main until it is finished. Your work sits on feature/new-checkout for three weeks, then merges in one go.

A feature flag lets the new code merge into main every day, deployed to production but switched off. An if around the new path reads a setting at runtime, and the setting decides who sees it: nobody, your team, 1% of users, everyone.

Both hide unfinished work. They differ in when your code meets everyone else's. With a branch, that happens once, at the end, all at once. With a flag, it happens every day, a little at a time.

Your answer

Your team is spread across Bengaluru and Berlin. Two engineers are each building a large feature over the next month in the same service. How would you keep main stable and still avoid a painful merge at the end?

Why long-lived branches hurt#

While your branch lives, main keeps moving. After three weeks, a busy team has merged hundreds of changes you have never run with yours. Merging is then a big conflict resolution session, in code other people changed for reasons you do not know, and the tests that pass on your branch say little about how it behaves on top of everything else. On a remote team this is worse: the person who wrote the conflicting change is asleep while you resolve it.

In 2010 Vincent Driessen published GitFlow, a popular model with long-lived develop and release branches. In 2020 he added a note to the top of his own post, saying that for web apps that are delivered continuously, he would recommend a much simpler workflow instead. GitFlow still fits software that ships numbered versions, like a library or a desktop app.

Trunk-based development#

The alternative is trunk-based development: everyone merges small changes into main (the trunk) at least daily, and branches live for hours or a day or two, never weeks. Google has worked this way in one huge repository for a long time. It is what makes CI mean something: you cannot integrate continuously with work hidden on a branch for a month.

Unfinished work stays hidden in one of two ways:

  • Branch by abstraction: put an interface in front of the old code, build the new version behind it piece by piece, then switch over. No flag service needed.
  • A feature flag: the new code path is in production but switched off, and you switch it on for more people as confidence grows.

Deploy is not release#

That is the real power of flags: deploying (putting code on servers) and releasing (letting users see it) become separate decisions. Deploys happen many times a day and nobody notices. Releases happen when product is ready, to whoever you choose, and undoing one is a switch, not a deploy.

Ship the same buggy checkout both ways and compare.

Launch a new checkout
Rollout of new-checkout
Customers on the new checkout, per hour
120
Failed payments per hour
7
Time to undo
about 10 seconds (flip the flag)
A small slice sees it. 7 failed payments an hour is enough to show up clearly on a dashboard split by flag value, and turning it off loses about 1 more. Find the bug here, not at 100%.
Scenario numbers: 12,000 checkouts an hour, and the bug fails 6% of new checkouts. Redeploy time and flag update time vary by team and tool.

Pete Hodgson's article on Martin Fowler's site sorts flags into four kinds, and they have very different lifetimes:

  • Release flags hide unfinished or untested work. Days to weeks, then delete.
  • Experiment flags split users for an A/B test. Live as long as the experiment.
  • Ops flags, or kill switches, turn off an expensive or risky feature under load ("disable recommendations if the database is struggling"). Can live for years, on purpose.
  • Permission flags turn features on for some users: a paid plan, beta testers, staff. Long lived, and really part of the product.

How flags are built#

Most teams use a flag service rather than a config file, because changing a config file means a deploy. The service holds the rules ("on for staff, and for 10% of users in India"), and an SDK in your app evaluates them. OpenFeature, a CNCF project, gives you one API in front of any vendor, so switching from one flag service to another changes one line.

ts
import { OpenFeature } from '@openfeature/server-sdk'
 
// The provider is the only vendor-specific line: LaunchDarkly, Unleash, Flagsmith...
await OpenFeature.setProviderAndWait(new YourFlagProvider({ sdkKey: process.env.FLAGS_KEY }))
const flags = OpenFeature.getClient()
 
export async function checkout(req) {
  const useNew = await flags.getBooleanValue('new-checkout', false, {
    targetingKey: req.user.id, // same user, same answer on every request
    country: req.user.country,
  })
  return useNew ? newCheckout(req) : oldCheckout(req)
}

Four details in that snippet matter in practice:

  1. The default (false) is what happens when the flag service is unreachable. Make it the safe, old behaviour.
  2. The targeting key keeps users sticky. A user hashed into the 10% stays there, instead of flickering between old and new checkout on every page.
  3. Evaluation is local. Server SDKs download the rules and keep them in memory, updated by a stream or a poll, so checking a flag costs microseconds, not a network call.
  4. Both paths are tested. While the flag exists, CI runs the important tests with it on and off, because production has both.
Flags are debt with a date on them

On 1 August 2012, Knight Capital deployed new trading code to eight servers, but one server was missed. The new code reused an old flag that, on the missed server, switched on trading logic that had been dead for years. In about 45 minutes it made millions of unintended trades and lost more than $460 million, by the SEC's count, and the firm did not survive as an independent company. Delete release flags when they reach 100%. Many teams give every flag an owner and an expiry date, and fail the build or open a ticket when it passes.

Names you will hear
  • LaLaunchDarklyThe best known hosted flag service
  • UnUnleashOpen source, self hosted or managed
  • OpOpenFeatureCNCF standard API in front of any flag vendor
  • PostHogFlags together with product analytics and experiments
  • GrGrowthBookOpen source flags and A/B testing
  • VercelFlags SDK for Next.js and edge middleware
ChoiceWhat you gainWhat you payPick it when
Long-lived branchSimple: no extra code paths, no flag service, unfinished work is physically separate.One big, risky merge at the end, and tests that never ran against the real main.Versioned releases (libraries, desktop apps), or a short spike you may throw away.
Trunk-based with feature flagsDaily integration, releases decoupled from deploys, instant kill switch and gradual rollout.Two code paths to test while the flag lives, a flag service to run or pay for, and flag debt if nobody cleans up.Web services and apps that deploy often. The default for most product teams.
Branch by abstractionReplaces a big component on trunk without a flag service, one small change at a time.Needs a clean interface around the old code first, which can be work in itself.Large rewrites inside the code: a new database layer, a new payment provider.
Which kind of flag?
Hide the half-built new search page until it is finished.
Turn off image previews if the image service is overloaded.
Show a green "Buy now" button to half the users and blue to the other half.
Only Pro plan customers can export to PDF.

In the interview#

Interviewer
How would you ship a risky rewrite of the checkout flow?
You
Trunk-based, behind a release flag. The new flow merges in small pull requests every day, switched off, so there is never a big merge. When it is complete I turn it on for staff, then 1% of users, then 10, 50 and 100, watching error rate and conversion split by flag value at each step. Users are hashed on their ID so they stay on one version. If anything looks wrong I turn it off, which takes seconds. Once it has been at 100% for a week or two, I delete the flag and the old code, because a flag left behind is how Knight Capital lost more than $460 million.
Daily merges, staged rollout with metrics, sticky targeting, kill switch, and cleanup.
Interviewer
What if the flag service goes down?
You
The SDK keeps the last rules it downloaded in memory, so evaluation carries on. If the app starts without ever reaching the service, every flag returns its default, so defaults must be the safe old behaviour. I would also alert on SDK connection errors so we know we are running on stale rules.
Knows evaluation is local and that defaults are a design decision.

Checkpoint#

Checkpoint

1. What does a feature flag separate?

2. A flag rolled out to 100% two months ago. What should happen?

3. Why pass the user ID as the targeting key in a percentage rollout?

Worth memorising
  • Deploy is not release. Flags make release a setting, and undo a switch.
  • Trunk-based: merge to main at least daily. Branches live for hours, not weeks.
  • Four kinds of flag: release, experiment, ops (kill switch), permission.
  • Default to the safe old path. Target on a stable user ID.
  • Knight Capital, 2012: an old flag, one missed server, more than $460 million in about 45 minutes. Delete flags.
Say this in 60 seconds

A long-lived branch hides unfinished work by keeping it out of main, so it meets everyone else's code once, at the end, in one big risky merge. Trunk-based development merges small changes into main every day and hides unfinished work behind a feature flag or behind an abstraction, which separates deploying code from releasing it. A flag service holds the rules and an SDK evaluates them locally in memory, with a stable user ID so users stay in one group, and a default that is the safe old behaviour if the service is unreachable. I roll out in stages, staff then one percent then wider, watching metrics split by flag value, and turning the flag off takes seconds. There are four kinds: release, experiment, ops kill switches and permission flags. Release flags get deleted soon after they reach a hundred percent, because old flags are dead code waiting to be switched back on.