Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Principles for building and scaling feature flag systems

docs.getunleash.io

Post reply on HN
Print view View original

61–70 of 115 posts

Re: Principles for building and scaling feature flag systems

#61

Post by bluefirebrand »

llbeansandrice wrote: post #59
gastonfournier wrote: post #27

Earlier quoted context omitted.

I faced something similar, and I think it's unavoidable. Give people a screwdriver and they'll find a way of using it as a hammer. The best you can do is expect the feature flagging solution to give some kind of warning for tech debt. Then equip them with alternative tools for configuration management. Rather than forbidding, give them options, but if it's not your scope, I'd let them be (I know as engineers this is…

> Give people a screwdriver and they'll find a way of using it as a hammer. I feel like feature flags aren't that far off though. They're fantastic for many uses of runtime configuration as mentioned in another comment. There's multiple people in this thread complaining about "abuse" of feature flags but no one has been able to voice why it's abuse instead of just use beyond esoteric dogma.

Allow me to try:

Feature Flags inherently introduce at least one branch into your codebase.

Every branch in your codebase creates a brand new state your code can run through.

The number of branches introduced by Feature Flags likely does not scale linearly, because there is a good chance they will become nested, especially as more are added.

Start with even an example of one feature flag nested inside another. That creates four possible program states. Four is not unreasonable, you can clearly define what state the program should be in for all four states.

Now scale that to a hundred feature flags, some nested, some not.

It becomes impossible to know what any particular program state should be past the most common configurations. If you can't point to a single interface in a program and tell me all of the possible states of it, your program is going to be brittle as hell. It will become a QA nightmare.

This is why Feature Flags should be used for temporary development efforts or A/B testing, and removed.

Otherwise you're going to have a debugging nightmare on your hands eventually.

Edit: Note that this is different from normal runtime configurations because normally runtime configurations don't have a mix of in-dev options and other temporary flags. Also, they aren't usually set up to arbitrarily add new options whenever it is convenient for a developer.

Re: Principles for building and scaling feature flag systems

#63

Post by erik_seaberg »

aranchelk wrote: post #14

With regard to web-based services, once you’ve got the ability to do canary testing, IMO flags/toggles are less compelling — busier code and logic you’ll have to pull out later.

Canarying gets you a 1/n treatment group, but it might be skew geographically (all affected users are near the canary’s datacenter). You need a percentage in a feature flag if 1/n is too big and you want, e.g., 0.1% of traffic.

Re: Principles for building and scaling feature flag systems

#64

Post by Dirak »

mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

I feel like this is a solvable problem: 1) make feature flags be configured to have an expiration date. If over the expiration date, auto-generate a task to clean up your FF 2) If you want to be extra fancy, set up a codemod to automatically clean up the FF once it's expired

I don't see the problem with developers using flags for configuration as a stopgap until there's a better solution available.

Re: Principles for building and scaling feature flag systems

#65

Post by hinkley »

mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

Long lived features flags is a development process bug, I'm not sure we can solve it with the feature toggle system.

I'm at the point of deciding that Scrum is fundamentally incompatible with feature flags. We demo the code long before the flag has been removed, which leads to perverse incentives. If you want flags to go away in a timely manner you need WIP limits, and columns for those elements of the lifecycle. In short: Kanban doesn't (have to) have this problem.

And even the fixes I can imagine like the above, I'm not entirely sure you can stop your bad actor, because it's going to be months before anyone notices that the flags have long overstayed their welcome.

I'm partial to flags being under version control, where we have an audit trail. However time and again what we really need is a summary of how long each flag has existed, so they can be gotten rid of. The Kanban solution I mention above is only a 90% solution - it's easy to forget you added a flag (or added 3 but deleted 2)

Re: Principles for building and scaling feature flag systems

#66

Post by hinkley »

baq wrote: post #31
mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

Or... see them for what they are: runtime configuration. The name implies a use case scenario, but in reality it's just a configuration knob. With a good UI, it's a pretty damn convenient way to do runtime configuration. So of course they'll be used for long-term configuration purposes, especially under pressure and for gradual rollouts of whole systems, not just A/B testing features.

Our FF system uses our config system as its system of record. There's some potential for misuse, and it's difficult to apply deadlines. On the plus side all our settings are captured in version control. Before they were spread out over several systems, one of which had an audit system that was pure tribal knowledge for years.

Re: Principles for building and scaling feature flag systems

#67

Post by baq »

bluefirebrand wrote: post #61
llbeansandrice wrote: post #59

Earlier quoted context omitted.

> Give people a screwdriver and they'll find a way of using it as a hammer. I feel like feature flags aren't that far off though. They're fantastic for many uses of runtime configuration as mentioned in another comment. There's multiple people in this thread complaining about "abuse" of feature flags but no one has been able to voice why it's abuse instead of just use beyond esoteric dogma.

Allow me to try: Feature Flags inherently introduce at least one branch into your codebase. Every branch in your codebase creates a brand new state your code can run through. The number of branches introduced by Feature Flags likely does not scale linearly, because there is a good chance they will become nested, especially as more are added. Start with even an example of one feature flag nested inside another. That c…

Sorry, not buying it.

Branches are difficult to reason about? Yes, I agree.

Are branches necessary to make the product behave in a different way in some circumstances? Most of the time.

Do those circumstances require a branch? Unless you’re super confident about some part of code, yes? But why would you be?

Runtime configuration is not about making QA easy. It’s introduced because QA has been hell already so you can control rollout of code which you know wasn’t properly QA’d - or it was but turns out the thing you built isn’t the thing users want and the release cycle is too long to deploy a revert.

I’d say ‘branches are bad but alternatives are worse’.

Re: Principles for building and scaling feature flag systems

#68

Post by baq »

zellyn wrote: post #9
zellyn wrote: post #6

Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause product…

Oh, and one last(?) update. If you create your own service to evaluate a bunch of feature flags for a given user/client/device/location/whatever and return the results, for use in mobile clients (everyone does this), PLEASE *make sure the client enumerates the list of flags it wants*. It's very tempting to just keep that list server-side, and send all the flags (much simpler requests, right?), but you will have to ke…

You should be collecting metrics on used flags and their values if you’re rolling your own. A saas offering will do that for you.

Re: Principles for building and scaling feature flag systems

#69

Post by baq »

aranchelk wrote: post #14

With regard to web-based services, once you’ve got the ability to do canary testing, IMO flags/toggles are less compelling — busier code and logic you’ll have to pull out later.

OTOH a flag gives you an ability to deploy and revert independent of the product’s release cycle.

Re: Principles for building and scaling feature flag systems

#70

Post by baq »

rubicon33 wrote: post #41
hn_throwaway_99 wrote: post #37

Earlier quoted context omitted.

This misses the point. A big point of feature flags is that you don't yet know how features will be perceived until you get them in front of real users. I get what you'd like "as an engineer", but it ignores the needs of the business.

Isn't that the job a product manager? There are other means and methodologies for gathering user sentiment before you go and build something. You should get as close as you can, release the product, and iterate. Todays world is release the product in some ramshackle form or fashion, collect feedback, iterate. To do that introduces a new construct of Feature Flags that would otherwise not be necessary.

Yeah product manager says ‘run this on 5% of traffic for 2 weeks and then we’ll see what the next iteration should be’.
Post reply on HN