Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When Feature Flags Do and Don't Make Sense (2019)

software.rajivprab.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: When Feature Flags Do and Don't Make Sense (2019)

#21

Post by deathanatos »

willsmith72 wrote: post #17
locknitpicker wrote: post #15

Earlier quoted context omitted.

> So your release which is "safer" behind a flag is actually untested until it's launched. It's only untested if your intention is to not test it. There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.

how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?

This sort of combinatorical complexity only really exists in theory. The overwhelmingly common case is that FF A and FF B are independent, and can be successfully tested independently. You test FF A on & off, FF B on & off; your tests grow linearly with your FF set (which you should be pruning as soon as the feature is deployed).

Also, since the original comment stated,

> often skips integration/e2e testing

They're probably responding with the expectation of this being compared to doing no testing at all. Merely assuming your FFs are independent, and testing them as such, is greater than nothing at all, even if the rare correlation might exist.

And where a correlation might exist, well, that's what your judgement & expertise is for. Absolutely test that if you find yourself in that case.

Re: When Feature Flags Do and Don't Make Sense (2019)

#22

Post by deathanatos »

sgarland wrote: post #7

As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week. Just use canaries, I’m begging you.

Depending on the exact specifics of what you mean by "canary", they have their own issues.

If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)

If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)

Re: When Feature Flags Do and Don't Make Sense (2019)

#23

Post by thraxil »

classictraffic wrote: post #2

I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved. I think the true cost is that feature flags can cause code bloat and readability issues, since eng…

> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.

I would argue that if flipping a feature flag isn't faster than doing a rollback, your feature flags are not functional. They should be essentially instantaneous and the first thing to reach for as soon as you've identified a problem.

> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.

This I totally agree with. Using feature flags well requires some discipline on the team's part to stay on top of them and remove them once they've served their purpose.

Re: When Feature Flags Do and Don't Make Sense (2019)

#24

Post by locknitpicker »

willsmith72 wrote: post #17
locknitpicker wrote: post #15

Earlier quoted context omitted.

> So your release which is "safer" behind a flag is actually untested until it's launched. It's only untested if your intention is to not test it. There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.

how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?

> how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?

You're going way out of your way to imagine problems where there are none. Feature flags are ephemeral and work as an ad-hoc release toggled at runtime. You create a feature flag, you commit the changes you need to commit behind the feature flag. Before switching the feature flag you run tests in preparation for the feature flag release, an finally you flip the flag. After that point, you either rollback the feature flag or you remove the feature flag.

This is not rocket science.

Re: When Feature Flags Do and Don't Make Sense (2019)

#25

Post by ollysb »

stopping wrote: post #18

I never found a great way to incorporate feature flags into my workflow without inducing significant mental churn managing 12-step rollouts over a dozen independent active flags. The changes I tend to make are sweeping, non-trivial refactors of base libraries with hundreds or possibly thousands of callers. These sorts of changes are exceptionally hard to flag (especially API changes), and it's stupidly easy for anoth…

What you're describing sounds like code hygiene and refactoring not features.

Re: When Feature Flags Do and Don't Make Sense (2019)

#26

Post by jasonpeacock »

jimbob45 wrote: post #20

If you’re using feature flags because you’re scared of rolling back, then your CI/CD pipeline is probably too brittle or complicated.

Or because rollbacks (and release) are slower than flipping a flag.

I’d rather invest in the feature flag upfront so I can unbreak customers in 2min rather than telling them to wait hours for the rollback.

Why are rollback and release so slow? Because we’re deploying to 40+ availability zones world-wide, totaling many thousands of servers, and we don’t want to worsen the impact by doing an accelerated “everything-at-once” rollback.

Re: When Feature Flags Do and Don't Make Sense (2019)

#27

Post by sgarland »

deathanatos wrote: post #22
sgarland wrote: post #7

As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week. Just use canaries, I’m begging you.

Depending on the exact specifics of what you mean by "canary", they have their own issues. If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while not…

> If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity

Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.

Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.

Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.

Re: When Feature Flags Do and Don't Make Sense (2019)

#28

Post by otterley »

conradludgate wrote: post #19

I've been introducing feature flags into our component at work. The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain. A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't de…

Out of curiosity, why can’t you change the behavior that prevents you from doing rollbacks?

This design seems rather brittle. What happens if a node spontaneously fails, or you need to move to another AZ? (Assuming you’re in the cloud.)

Re: When Feature Flags Do and Don't Make Sense (2019)

#29

Post by otterley »

locknitpicker wrote: post #16
lgrthmsprs wrote: post #4

That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors. And that brings us to another gr…

> That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. T I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software. Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix a…

Both Google and Amazon do rollbacks instead of feature flag switches. I’m pretty sure they have some experience maintaining user-facing software.

Re: When Feature Flags Do and Don't Make Sense (2019)

#30

Post by stopping »

ollysb wrote: post #25
stopping wrote: post #18

I never found a great way to incorporate feature flags into my workflow without inducing significant mental churn managing 12-step rollouts over a dozen independent active flags. The changes I tend to make are sweeping, non-trivial refactors of base libraries with hundreds or possibly thousands of callers. These sorts of changes are exceptionally hard to flag (especially API changes), and it's stupidly easy for anoth…

What you're describing sounds like code hygiene and refactoring not features.

In my old org the policy was to flag every change, not just new features and behaviors. I'd even argue that refactoring is often a higher risk than adding a new feature path, because the scope of a refactor can touch nearly every user journey. So I can understand why the policy applies to refactoring. I'm just lamenting that this doesn't seem to be well-trodden ground, and being a person who cares deeply about reducing system complexity makes this whole thing kind of a bummer.
Post reply on HN