Earlier quoted context omitted.
> I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change and using an (often expensive) feature flags as a service platform: roll your own simple system. The middle ground is a JSON file that is copied up and periodically refreshed. We (Sentry) moved from a managed software to just a YAML file with feature flags that is pushed to all containers…
How do you push the files to all of your containers? I’ve done this in the past with app specific endpoints but never found a solution I liked with containers.
It's OK to hardcode feature flags
51–60 of 111 posts
Re: It's OK to hardcode feature flags
#52Earlier quoted context omitted.
Seriously. This is one of those cases where rolling your own really does make sense. Flags in a DB table, flags in a json file, all super simple to build and maintain, and 100x faster and more reliable than making the critical paths of your application's request cycle depend on an external provider.
In years of trying to sell things I've found that one of the best selling points to management is "susceptible to vendor lock-in", "you don't own your customer database", etc. I have no idea why that is.
Re: It's OK to hardcode feature flags
#53The single biggest value add of feature flags is that they de-risk deployment. They make it less frightening and difficult to turn features on and off, which means you'll do it more often. This means you can build more confidently and learn faster from what you build. That's worth a lot . I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change a…
> using an (often expensive) feature flags as a service platform I have no idea why anyone would actually do that in real life. Feature flags are something so trivial that you can implement them from scratch in a few hours, tops — and that includes some management UI.
We have a FF as a service platform and a big "value add" is that we can turn on and off features at the client level with it.
But, unfortunately, it's both not the only mechanism for this and it is also being used for actual feature flags and not just client specific configuration.
I'm personally a MUCH bigger fan of putting feature flags in a configuration file that you deploy either with the application or though some mechanism like kubernetes configs. It's faster, easier to manage, and really easy to quickly answer the question "What's turned on, why, and for how long". Because, a core part of managing feature flags is deleting them and the old code path once you are confident things are "right".
The biggest headache of our FF-ws is that's really not clear and we OFTEN end up with years old feature flags that are on with the old code path still existing even though it's unexercised.
Re: It's OK to hardcode feature flags
#54Earlier quoted context omitted.
> build-vs-buy Roll your own. Seriously. Feature flags are such an easy thing that there should be a robust and completely open source offering not tied to B2B SaaS. Until then, do it in house. My team built a five nines feature flag system that handled 200k QPS from thousands of services, active-active, local client caching, a robust predicate DSL for matching various conditions, percent rollout, control plane, ACLs…
I don't understand what a dedicated "completely open source offering" provides or what your "five nines feature flag system" provides. If you're running on a simple system architecture, then you can sync some text files around, and if you have a more scalable distributed architecture, then you're probably already handling some kind of slowly-changing, centrally-managed system state at runtime (e.g. authentication/aut…
His point was that even a feature flag system in a complex environment with substantial functional and system requirements is worth building vs buying. If your needs are even simpler, then this statement is even more true!
I'm having a hard time making sense out of the rest of your comment, but in larger businesses the kinds of things you're dealing with are:
- low latency / staleness: You flip a flag, and you'll want to see the results "immediately", across all of the services in all of your datacenters. Think on the order of one second vs, say 60s.
- scalability: Every service in your entire business will want to check many feature flags on every single request. For a naive architecture this would trivially turn into ungodly QPS. Even if you took a simple caching approach (say cache and flush on the staleness window), you could be talking hundreds of thousands of QPS across all of your services. You'll probably want some combination of pull and push. You'll also need the service to be able to opt into the specific sets of flags that it cares about. Some services will need to be more promiscuous and won't know exactly which flags they need to know in advance.
- high availability: You want to use these flags everywhere, including your highest availability services. The best architecture for this is that there's not a hard dependency on a live service.
- supports complex rules: Many flags will have fairly complicated rules requiring local context from the currently executing service call. Something like: "If this customer's preferred language code is ja-JP, and they're using one of the following devices (Samsung Android blah, iPhone blargh), and they're running versions 1.1-1.4 of our app, then disable this feature". You don't want to duplicate this logic in every individual service, and you don't want to make an outgoing service call (remember, H/A), so you'll be shipping these rules down to the microservices, and you'll need a rules engine that they can execute locally.
- supports per-customer overrides: You'll often want to manually flip flags for specific customers regardless of the rules you have in place. These exclusion lists can get "large" when your customer base is very large, e.g. thousands of manual overrides for every single flag.
- access controls: You'll want to dictate who can modify these flags. For example, some eng teams will want to allow their PMs to flip certain flags, while others will want certain flags hands off.
- auditing: When something goes wrong, you'll want to know who changed which flags and why.
- tracking/reporting: You'll want to see which feature flags are being actively used so you can help teams track down "dead" feature flags.
This list isn't exhaustive (just what I could remember off the top of my head), but you can start to see why they're an endeavor in and of themselves and why products like LaunchDarkly exist.
Re: It's OK to hardcode feature flags
#55Earlier quoted context omitted.
> using an (often expensive) feature flags as a service platform I have no idea why anyone would actually do that in real life. Feature flags are something so trivial that you can implement them from scratch in a few hours, tops — and that includes some management UI.
Often these 3rd party offerings are feature flags PLUS experimentation with user segmenting. Depending on the style of software you build, this can be extremely valuable; it’s very popular in the SaaS market for a reason. Early on at Notion we used simple percent rollout in Redis, then we built our own flag & experimentation system, but as our needs got more complex we ended up switching to a 3rd party rather than de…
Re: It's OK to hardcode feature flags
#56Earlier quoted context omitted.
When your app starts to get bigger and more complex, the idea of needing to restart a process to pick up any new kind of data starts to seem silly. Have seen the pattern many times: Hard-code values in code -> configure via env -> configure slow things via env and fast things via redis -> configure almost everything via a config management system I do not want to reboot every instance in a fleet of 2000 nodes just to…
How do you do software upgrades if you don't have a good system for handling process restarts without downtime?
At my last job, updating a productive game server cluster took an hour or so with minimal to no customer interruption. Though you could still see and measure how the systems needed another hour or two to get their JIT'ers, database caches, code caches and all of these things back on track. Maybe you can just say "then architect better" or "just use rust instead of Java", but the system was as it was and honestly, it performed vey very well.
On the other hand, the game servers checked once a minute what promotion events should be active every minute from the marketing backend and reacted to it without major caching/performance impacts.
Similar things at my current place. Teams have stable and reliable deployment mechanisms that can bring code to Prod in 10 - 15 minutes, including rollbacks if necessary. It's still both safer to gate new features behind feature toggles, and faster to turn feature toggles on and off. Currently, such per-customer configs apply in 30 - 60 seconds across however many applications deem it relevant.
I would have to think quite a bit to bring binaries to servers that quickly, as well as coordinate restarts properly. The latter would dominate the time easily.
Re: It's OK to hardcode feature flags
#57Earlier quoted context omitted.
> build-vs-buy Roll your own. Seriously. Feature flags are such an easy thing that there should be a robust and completely open source offering not tied to B2B SaaS. Until then, do it in house. My team built a five nines feature flag system that handled 200k QPS from thousands of services, active-active, local client caching, a robust predicate DSL for matching various conditions, percent rollout, control plane, ACLs…
I don't understand what a dedicated "completely open source offering" provides or what your "five nines feature flag system" provides. If you're running on a simple system architecture, then you can sync some text files around, and if you have a more scalable distributed architecture, then you're probably already handling some kind of slowly-changing, centrally-managed system state at runtime (e.g. authentication/aut…
At scale the nines of your feature flagging system become the nines of your company.
We have a massive distributed systems architecture handling billions in daily payment volume, and flags are critical infra.
Teams use flags for different things. Feature rollout, beta test groups, migration/backfill states, or even critical control plane gates. The more central a team's services are as common platform infrastructure, the more important it is that they handle their flags appropriately, as the blast radius of outages can spiral outwards.
Teams have to be able to competently handle their own flags. You can't be sure what downstream teams are doing: if they're being safe, practicing good flag hygiene, failing closed/open, keeping sane defaults up to date, etc.
Mistakes with flags can cause undefined downstream behavior. Sometimes state corruption (eg. with complicated multi-stage migrations) or even thundering herds that take down systems all at once. You hope that teams take measures to prevent this, but you also have to help protect them from themselves.
> slowly-changing, centrally-managed system state at runtime
With flags being so essential, we have to be able to service them with near-perfect uptime. We must be able to handle application / cluster restart and make sure that downstream services come back up with the correct flag states for every app that uses flags. In the case of rolling restarts with a feature flag outage, the entire infrastructure could go hard down if you can't do this robustly. You're never given the luxury of knowing when the need might arise, so you have to engineer for resiliency.
An app can't start serving traffic with the wrong flags, or things could go wrong. So it's a hard critical dependency to make sure you're always available.
Feature flags sit so closely to your overall infrastructure shape that it's really not a great idea to outsource it. When you have traffic routing and service discovery listening to flags, do you really want LaunchDarkly managing that?
Re: It's OK to hardcode feature flags
#58The single biggest value add of feature flags is that they de-risk deployment. They make it less frightening and difficult to turn features on and off, which means you'll do it more often. This means you can build more confidently and learn faster from what you build. That's worth a lot . I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change a…
Our config files are stored in their own repo. Pushes to the master branch trigger a Jenkins job that copies the config files to a GCP bucket.
On startup, each machine pulls this config from GCS and everything just works.
It's not a 'redeployment' in the sense that we don't push new images on each config change.
Re: It's OK to hardcode feature flags
#59The single biggest value add of feature flags is that they de-risk deployment. They make it less frightening and difficult to turn features on and off, which means you'll do it more often. This means you can build more confidently and learn faster from what you build. That's worth a lot . I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change a…
I agree with a lot of this, except for the part about de-risking deployments. That should not be a reason why to adopt a feature flag platform - that is a symptom of a bad deployment pipeline that should be fixed which is a whole other story.
Re: It's OK to hardcode feature flags
#60The single biggest value add of feature flags is that they de-risk deployment. They make it less frightening and difficult to turn features on and off, which means you'll do it more often. This means you can build more confidently and learn faster from what you build. That's worth a lot . I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change a…
> using an (often expensive) feature flags as a service platform I have no idea why anyone would actually do that in real life. Feature flags are something so trivial that you can implement them from scratch in a few hours, tops — and that includes some management UI.
Permanent per customer configuration is not a feature flag. Also best would be not to have too many per customer configurations.