Principles for building and scaling feature flag systems • Hacker News Board

Re: Principles for building and scaling feature flag systems

#71

Post by monknomo » Fri, Sep 22, 2023, 9:28 PM UTC

bertil wrote: post #46

baq wrote: post #40

Earlier quoted context omitted.

I can immediately see if the config is being requested, which system requests it, what are the metadata of the request, etc. I can do conditional rollout of a configuration based on runtime data. I can reset the configuration to a know-good failsafe default without asking for approval with a break-glass button. I can schedule a rollout and get a reviewer for the config change. IME the feature flag interface is next t…

I am tempted to agree: if separating the two is key (I’m not convinced that it is, but happy to assume) why not copy the interface and the infrastructure of the feature flag and offer it as a configuration tool. I feel like you could easily add a status to flags, to mark whether they are part of a release process, or a permanent configuration tool, and in the latter case, take them off the release interfaces.

I think unleash offers "toggle type" which can take values that describe whether it's a pure feature flag, something operation, config, etc.

Re: Principles for building and scaling feature flag systems

#72

Post by dasil003 » Fri, Sep 22, 2023, 9:29 PM UTC

rubicon33 wrote: post #35

baq wrote: post #31

Earlier quoted context omitted.

Or... see them for what they are: runtime configuration. The name implies a use case scenario, but in reality it's just a configuration knob. With a good UI, it's a pretty damn convenient way to do runtime configuration. So of course they'll be used for long-term configuration purposes, especially under pressure and for gradual rollouts of whole systems, not just A/B testing features.

This hits the nail on the head. The term "feature flag" has come to inherently have a time component because features are supposed to eventually be fulled GA'd. What I've seen in practice is feature flags are never removed so a better way to think about them is as a runtime configuration.

I think the reason feature flags are never removed is because the timeframe that a given feature-flag is top-of-mind is also when it's at its most useful. Later when it's calcified in place and the off-state may be broken/atrophied, no one is really thinking about it.

I'm also not convinced it's always a huge problem. I can imagine sometimes it is, but in most codebases I've worked on, it's more of an annoyance but not cracking the top 3 or 5 biggest problems we wanted to focus on.

IMHO the best solution is not something heavy handed like a policy that we only use run-time config for fixed timeframes, or a process where we regularly audit and prune old flags. It's simply to keep a record of the config changes over time so anyone interested can see the history, and a culture where every engineer is encouraged to take a little extra time to verify and remove dead stuff whenever it crosses their path .

Re: Principles for building and scaling feature flag systems

#73

Post by strken » Fri, Sep 22, 2023, 9:53 PM UTC

mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

I think it's better to admit they actually are config, just a different kind of config that comes with an expiration date.

Accepting reality in this way means you'll design a config management system that lets you add feature flags with a required expiration date, and then notifies you when they're still in the system after the deadline.

Re: Principles for building and scaling feature flag systems

#74

Post by eximius » Fri, Sep 22, 2023, 10:43 PM UTC

The system we're building now meets most of these but not necessarily in the way described.

First, we're building a runtime configuration system on top of AWS AppConfig. YAML/proto validation that pushes to AppConfig via gitops and bazel. Configurations are namespaced so the unique names is solved. It's all open in git.

Feature flags are special cases of runtime configuration.

We are distinguishing backend feature flags from experimentation/variants for users. We don't have (or want) cohorting by user IDs or roles. We have a separate system for that and it does it well.

The last two points - distinguishing between experimentation/feature variants and feature flags as runtime configuration are somewhat axiomatic differences. Folks might disagree but ultimately we have that separate system that solves that case. They're complimentary and share a lot of properties but ultimately it solves a lot of angst if you don't force both to be the same tool.

Re: Principles for building and scaling feature flag systems

#75

Post by x0x0 » Fri, Sep 22, 2023, 11:44 PM UTC

llbeansandrice wrote: post #59

gastonfournier wrote: post #27

Earlier quoted context omitted.

I faced something similar, and I think it's unavoidable. Give people a screwdriver and they'll find a way of using it as a hammer. The best you can do is expect the feature flagging solution to give some kind of warning for tech debt. Then equip them with alternative tools for configuration management. Rather than forbidding, give them options, but if it's not your scope, I'd let them be (I know as engineers this is…

> Give people a screwdriver and they'll find a way of using it as a hammer. I feel like feature flags aren't that far off though. They're fantastic for many uses of runtime configuration as mentioned in another comment. There's multiple people in this thread complaining about "abuse" of feature flags but no one has been able to voice why it's abuse instead of just use beyond esoteric dogma.

The fundamental diff between feature flags and config is the former is meant to be a soft deploy of code where everyone is expected to eventually be on the new code. Thus it should have a timer built in where it stops, and you should consider all new customers launching with it on.

As for why: if you don't deprecate the feature flag in some time span, you're permanently carrying both code paths. With ongoing associated dev and qa resources and costs against your complexity budget.

Permanent costs should only be undertaken after careful consideration, and should be outside the scope of a single dev deciding to undertake them. Whereas flags should be cheap to add to enable dev to get stuff into prod faster while retaining safety.

Permanently making something a config choice should be done after heavier deliberation because of the aforementioned costs, and you often want different tools to manage it. Including something heavier duty than a single checkbox/button in your internal CS admin tooling. These are often tied into contracts or legal needs, and in many cases salesforce should be the source of truth for them. Or whatever CPQ system you're using.

Re: Principles for building and scaling feature flag systems

#76

Post by joshuamorton » Sat, Sep 23, 2023, 12:18 AM UTC

athenot wrote: post #16

daigoba66 wrote: post #11

Earlier quoted context omitted.

> One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant). Can you elaborate on this? As a programmer, I would think that using something like a constant would help us find references and ensure all usage of the flag is removed when the constant is removed.

Not OP but multiple code bases may refer to the same flag by a different constant. Having a single string that can be searched accross all repos in an organization is quite handy to find all places where it's referenced.

especially when you have different languages with different rules, `MY_FEATURE_FLAG` and `kMyFeatureFlag` and `@MyFeatureFlag` might all be reasonable names for what is defined as `"my_feature_flag"` in the configuration.

Using just the string-recognizable name everywhere is...better.

Re: Principles for building and scaling feature flag systems

#77

Post by bluefirebrand » Sat, Sep 23, 2023, 12:36 AM UTC

baq wrote: post #67

bluefirebrand wrote: post #61

Earlier quoted context omitted.

Allow me to try: Feature Flags inherently introduce at least one branch into your codebase. Every branch in your codebase creates a brand new state your code can run through. The number of branches introduced by Feature Flags likely does not scale linearly, because there is a good chance they will become nested, especially as more are added. Start with even an example of one feature flag nested inside another. That c…

Sorry, not buying it. Branches are difficult to reason about? Yes, I agree. Are branches necessary to make the product behave in a different way in some circumstances? Most of the time. Do those circumstances require a branch? Unless you’re super confident about some part of code, yes? But why would you be? Runtime configuration is not about making QA easy. It’s introduced because QA has been hell already so you can…

There is nothing worse than code with dozens to hundreds of possible configuration states that you must test for every new feature.

If your QA was bad before, you've made it worse.

"I can toggle it off without pushing a new release" is a terrible bandaid for the problem.

Re: Principles for building and scaling feature flag systems

#78

Post by brightball » Sat, Sep 23, 2023, 12:46 AM UTC

mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

It’s one of the main reasons to start with something like unleash because they have stale flag warnings built in. Plus, since you already have a UI it’s harder for it to be hijacked.

Re: Principles for building and scaling feature flag systems

#79

Post by zellyn » Sat, Sep 23, 2023, 12:54 AM UTC

hamandcheese wrote: post #52

adasdasdas wrote: post #45

More principles - Require in code defaults for fault tolerance - Start annoying the flag author to delete if the flag is over a month old - Partial rollout should be by hash on user id - Contextual flag features should always be supplied by client (e.g. only show in LA, the location should be provided by client)

> Partial rollout should be by hash on user id With a per-flag salt as well, otherwise the same user will always have bad luck and be subject to experiments first.

Our in-house solution hashes flagname+key, and LD does the same but adds salt

Re: Principles for building and scaling feature flag systems

#80

Post by charrondev » Sat, Sep 23, 2023, 1:11 AM UTC

hamandcheese wrote: post #50

zellyn wrote: post #25

Earlier quoted context omitted.

Sadly, this is a battle you are destined to lose. I have almost completely given up. The best you can aim for is to use feature flags better rather than worse. - Some flags are going to stay forever: kill switches, load shedding, etc. (vendors are starting to incorporate this in the UI) - Unless you have a very-easy-to-use way to add arbitrary boolean feature toggles to individual user accounts (which can become its…

Thank you for this great list of the immense business value derived from "misusing" feature flags!

In my org, I think I’ve go the feature flag thing mostly down.

We started with a customer specific configuration system that allows arbitrary values matching a defined schema. It’s very easy to add to the schema (define the config name, types, and permissions to read or write it in a JSON schema document).

We have an administration panel with a full view of the JSON config for our support specialist and and even more detailed one for developers.

Most config values get a user interface as well.

From there we just have a namespace in the configuration for “feature flags”. Sometimes these are very short lived (2-4 sprints until the feature is done), but others can last a lot longer.

There are an unfortunate couple that will probably never go away at this point (because of some enterprise customer with a niche use case in the “legacy” version of the feature that we’ve not yet implemented compatibility with and I don’t know when it will get on our roadmap to do so), but in the end they can just be migrated into normal config values if needed.

A little tooling layer on top lets us query and write to the configs of thousands of sites at once as well.