Ask HN: Does your team use feature flags?

One (product A) does individual integrations from the production feature flag system to the repo when the flag is at 100% in production. It’s quite successful.

The other (product B) does bulk integration every few hours from production to the repo, bringing all flags at once. It enables flags that have been “approved” (as opposed to fully enabled in production). It’s less successful because of the bulk behavior, which makes it more difficult to isolate breaks.

Both of these integrations go through the same validation gates as pull requests (they are actually implemented as pull requests). This ensures that flag changes through this system cannot break pull requests.

The ideal system for protecting pull requests (or whatever other validation) is to require validation to pass before the flag can be enabled in production. I have not implemented this as a hard gate, but I do have an “fyi” validation run as post of the flag enabling for product A so engineers see if they’ve broken something before they start turning on the flag.

Getting philosophical, I strongly believe that feature flags should not be enabled by default in the validation environment until they are fully enabled (or close to fully enabled) in production. This ensures that the base state (flag off) is validated. You always want it to be safe to turn off the flag, which means you want validation running with the flag off. Product B does not have this behavior for legacy reasons but I think it’s a poor technical decision.

   if FEATURE_FLAG_ENABLED():
       do_new_version()
   else:
       do_old_version()

Let's give it 1/100 bad code change, and 100 devs. The probability of a clean push is:

(1-1/100)^100 = 0.36. That's worse than a coin flip.

for 10 devs = 0.90. Still not great, one push in 10 will be bad.

When you're trying to push daily (or even more frequently), this will kill the team's velocity. It's even worse when there are interleaving changes making rollback and re-release impossible before the next release. Rollback and freeze is a problem when you're trying to meet deadlines (either marketing or regulatory).

Feature flags allow the code to go in and each change to be rolled back independently. This lets the rest of the team to make progress while the bug is debugged.

Short release pipelines could be a solution, but they're aren't necessarily sufficient. Time to recovery when there isn't a feature flag is also problem.

For example, some bugs take days/weeks to reveal themselves (think month end), so a "30 min build and full push" is not enough of a guarantee. Time to recovery needs to consider the time to find the change and roll it back, including dealing with any interleaving changes that happened in the meantime.

Our React app just uses a simple home-rolled set of keys in localStorage; we have a "secret" route that lets users turn their own flags on or off, and we encourage developers to duplicate big chunks of code while working behind a flag instead of mixing old and new, to make cleanup easier.

(And yeah, someone does have to stomp around every few months and remind everyone to clean up their dead code again. Still worth it.)