Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When Feature Flags Do and Don't Make Sense (2019)

software.rajivprab.com

Post reply on HN
Print view View original

31–38 of 38 posts

Re: When Feature Flags Do and Don't Make Sense (2019)

#31

Post by otterley »

locknitpicker wrote: post #14
llIIllIIllIIl wrote: post #3

I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software. 1 feature flag = 2 behaviors 2 feature flags = 4 behaviors So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.

> I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software. This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha? And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings? > So cogniti…

Some things that initially seem like benefits at a small scale become less so at large scale. Owning one cat can be lots of fun. Owning 50 cats, not so much.

Everyone intends to collect the garbage. Few actually do.

Re: When Feature Flags Do and Don't Make Sense (2019)

#32

Post by conradludgate »

otterley wrote: post #28
conradludgate wrote: post #19

I've been introducing feature flags into our component at work. The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain. A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't de…

Out of curiosity, why can’t you change the behavior that prevents you from doing rollbacks? This design seems rather brittle. What happens if a node spontaneously fails, or you need to move to another AZ? (Assuming you’re in the cloud.)

The design is postgres. It's our customers that do stateful things on those connections. Unfortunately it's tools like pgdump/pgrestore that are not tolerant to connection interruption.

Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically.

We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so it's easier to update inplace

Re: When Feature Flags Do and Don't Make Sense (2019)

#33

Post by otterley »

conradludgate wrote: post #32
otterley wrote: post #28

Earlier quoted context omitted.

Out of curiosity, why can’t you change the behavior that prevents you from doing rollbacks? This design seems rather brittle. What happens if a node spontaneously fails, or you need to move to another AZ? (Assuming you’re in the cloud.)

The design is postgres. It's our customers that do stateful things on those connections. Unfortunately it's tools like pgdump/pgrestore that are not tolerant to connection interruption. Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically. We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so…

What does it mean to do “stateful things on a connection”? Feel free to point me to literature if you’d be repeating something already written or well known.

Re: When Feature Flags Do and Don't Make Sense (2019)

#34

Post by locknitpicker »

otterley wrote: post #29
locknitpicker wrote: post #16

Earlier quoted context omitted.

> That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. T I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software. Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix a…

Both Google and Amazon do rollbacks instead of feature flag switches. I’m pretty sure they have some experience maintaining user-facing software.

> Both Google and Amazon do rollbacks instead of feature flag switches.

I'm sorry, but you're completely wrong. I know for a fact that Amazon uses extensively a A/B testing system for feature flags. Others in this discussion already mentioned it.

Re: When Feature Flags Do and Don't Make Sense (2019)

#35

Post by otterley »

locknitpicker wrote: post #34
otterley wrote: post #29

Earlier quoted context omitted.

Both Google and Amazon do rollbacks instead of feature flag switches. I’m pretty sure they have some experience maintaining user-facing software.

> Both Google and Amazon do rollbacks instead of feature flag switches. I'm sorry, but you're completely wrong. I know for a fact that Amazon uses extensively a A/B testing system for feature flags. Others in this discussion already mentioned it.

I’m not wrong. They do use feature flags, but not as a substitute for rolling back failed or buggy deployments. They (and Google too) use feature flags in the ways according to the best practices discussed in the article. Google also covers this well in their highly-regarded Site Reliability Engineering book.

Source: worked at AWS for 7 years.

And knock it off with the attitude.

Re: When Feature Flags Do and Don't Make Sense (2019)

#36

Post by willsmith72 »

deathanatos wrote: post #21
willsmith72 wrote: post #17

Earlier quoted context omitted.

how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?

This sort of combinatorical complexity only really exists in theory. The overwhelmingly common case is that FF A and FF B are independent, and can be successfully tested independently. You test FF A on & off, FF B on & off; your tests grow linearly with your FF set (which you should be pruning as soon as the feature is deployed). Also, since the original comment stated, > often skips integration/e2e testing They're p…

[dead]

Re: When Feature Flags Do and Don't Make Sense (2019)

#37

Post by willsmith72 »

locknitpicker wrote: post #24
willsmith72 wrote: post #17

Earlier quoted context omitted.

how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?

> how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es? You're going way out of your way to imagine problems where there are none. Feature flags are ephemeral and work as an ad-hoc release toggled at runtime. You create a feature flag, you commit the changes you need to commit behind the feature flag. Before switching the feature flag you run tests in pr…

just replied above. i wish it was my imagination, it's reality

Re: When Feature Flags Do and Don't Make Sense (2019)

#38

Post by deathanatos »

sgarland wrote: post #27
deathanatos wrote: post #22

Earlier quoted context omitted.

Depending on the exact specifics of what you mean by "canary", they have their own issues. If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while not…

> If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability. Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to crea…

I'm also an SRE¹. A lot of that does match with my experience. Esp. "I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability." — neither have I, I guess; perhaps I'm assuming that rapid iteration and stability are possible, whereas you seem to not. (And the evidence I've seen does suggest you might be right; I've certainly not found a way.)

> Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.

I almost mentioned automated stepped rollouts. I guess it's more because there is a staged rollout, non-technical types inexorably inject bureaucracy and manual "approvals" into such things.

I think the other problem I'm faced as an SRE with automated stepped rollouts is "they take too long" and slow down velocity; which, I guess again, you don't see as problematic. I guess I don't know how to "market" that to the SWEs I work with. Even a non-staged k8s Deployment at my last job was frustrating devs due to the time it would take to roll out, even with no real steps, aside from "roll out new version", but the workload itself was so slow that that would still take significant time. (But nobody was willing to invest into "fix workload startup and shutdown times" — some huge fraction of which I think was just loading Google's Vertex library, IIRC, which is … a sign of the times.)

¹ty for your reply, too; that was … damning? skewering?, but insightful.

Post reply on HN