It's OK to hardcode feature flags

> It's even okay to hardcode them into code (not a config/json file).

Yes, as I understood it that was what the article was all about.

> One sense of feature flag that I am familiar with (not from experience) is in trunk based development where they are used to integrate new code (with a feature flag) which is relatively untested. Or just not fully developed. That’s an alternative to longer-lived feature branches which are only merged until it is either fully finished or (going further) fully tested.

That's actually the only sense of "feature flag" I was aware of before this discussion.

> Hard-coding that kind of feature flag makes sense. Because a later revision will delete the feature flags outright (removing the branches).

Yup. And, AFAIK, is what "feature flag" means.

> There also seems to be feature flags in the sense of toggling on and off features. Then hard-coding makes less sense.

So "feature flag" has now taken on -- taken over? -- the meaning of just plain "flag" (or "switch" or "toggle" or whatever), as in ordinary everyday run-time configuration? What is this development supposed to be good for? We used to have two distinct distinguishable terms for two distinct distinguishable things; now we apparently don't any more. So we've lost a bit of precision from the language we use to discuss this stuff. Have we, in exchange, gained anything?

It seems like you've kind of missed both of my points.

If you're doing canary deploys to a fleet of 2000 nodes, it might take hours for the config to make it to all of them (I've seen systems where a fleet upgrade can take a week to make it all the way out). If your feature flags are configured that way, there's a long time that the state of a flag will be in that in-between state. We put feature flags in the database not config/environment so that we can turn a feature on or off more or less atomically. Ie, an admin goes into the management interface, flips a flag from off to on and then every single request that the system serves after that reflects that state. As long as you're using a database that supports transactions, you absolutely can have a clear point in time that delineates before/after that change. Rolling out a config change to a large fleet, you don't get that.

On the second point, what I'm saying is that (talk to your friendly local SRE if you don't believe me), a large percentage of production incidents in large systems are because of configuration changes, not application changes. This is because those things are significantly harder to really test than application code. Eg, if someone sets an environment variable for the production environment like `REDIS_IP=10.0.0.13` how do you know that's the correct IP address in that environment? You can add a ton of linting, you can do reviews, etc, but ultimately, it's a common vector for mistakes and it's one of the hardest areas to completely prevent human error from creating a disaster. One of the best strategies we have is to structure the system so you don't have to make manual environment/config changes that often. If you implement your feature flag system with environment variables/config, you'll be massively increasing the frequency that people are editing and changing that part of the system, which increases the chances of somebody making a typo, forgetting to close a quote, missing a trailing comma in a json file, etc.

Where I work we make production config changes maybe once a week or so and it's done by people who know the infrastructure very well, there's a bunch of linting and validation, and the change is rolled out with a canary system. In contrast, feature flags are in the database and we have a nice, very safe custom UI so folks on the Product and Support teams can manage the flags themselves, turning them on/off for different customers without having to go through an engineer; they might toggle flags a dozen times a day.

There is something too this, though jumping all the way to DIY is unnecessary.

Context: I run a FF company (https://prefab.cloud/)

There are multiple distinct benefits to be had from feature flagging. Because it's the "normal" path, most FF products bundle them all together, but it's useful to split them out.

- The code / libraries for evaluating rules. - The UI for creating rules, targeting & roll outs. - The infrastructure for hosting the flags and providing real-time updates. - Evaluation tracking / debugging to help you verify what's happening.

If you don't need #1 and #2 there, you might decide to DIY and build it yourself, but I think you shouldn't have to. Most feature flag tools today are usable in an offline mode. For Prefab it is: https://docs.prefab.cloud/docs/how-tos/offline-mode You can just do a CLI command to download the flags. Then boot the client off a downloaded file. With our pricing model that's totally free because we're really hardly doing anything for you. Most people use this functionality for CI environments, but I think it's a reasonable way to go for some orgs. It has 100% reliability and that's tough to beat.

You can do that if you DIY too, but there's so many nice to haves in actually having a tool / UI that has put some effort into it that I would encourage people not to go down that route.

That sounds like bad engineering.

Having all kinds of flags makes system prone to misconfigurations.

As amount grows you get flags depending on other flags etc.

Gets insane rather quickly. Unless someone purges flags relentlessly.