11–20 of 38 posts
As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week. Just use canaries, I’m begging you.
There is some confusion in this post. For risk management, feature flags are used to mitigate the risk of issues not being caught by the QA process. Wishing for a QA process to be 100% flawless doesn't make it so.
Also, guess what an incremental deployment is.
I think the blogger tried to make sweeping statements but didn't actually spent any time thinking through the arguments.
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved. I think the true cost is that feature flags can cause code bloat and readability issues, since eng…
Not quite. When you add a feature flag, you now have a system that has N separate code flows that you need to verify before launch,and then have to modify and retest to remove the feature flag. A feature flag ends up doubling the workload to test and verify the feature.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This is perhaps the most relevant point that the blogger misses. Feature flags are runtime switches that can extend simultaneously to multiple systems and even clients. You can't pull features from clients with a release, where week-long release processss can barely get you a 60% uptake. Redeploying away features in the backend can easily take half an hour, specially if it's a system that does multi region rolling deployments.
If you want instant switches, your best option is switches flipped at runtime. That's what a feature flag is.
I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software. 1 feature flag = 2 behaviors 2 feature flags = 4 behaviors So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha?
And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings?
> So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
Not really. Feature flags have a life cycle which is managed by release managers, and the goal is to always get rid of them asap. They are introduced to manage the risk of a rollout, and when the transition is over you pull the feature flags. Done.
Another issue is launching behind feature flags often skips integration/e2e testing, which couldn't possibly test with every combination of every flag. So your release which is "safer" behind a flag is actually untested until it's launched.
It's only untested if your intention is to not test it.
There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors. And that brings us to another gr…
I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software.
Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature.
Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved.
In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
Another issue is launching behind feature flags often skips integration/e2e testing, which couldn't possibly test with every combination of every flag. So your release which is "safer" behind a flag is actually untested until it's launched.
> So your release which is "safer" behind a flag is actually untested until it's launched. It's only untested if your intention is to not test it. There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
To this day I've never found any good guidelines for flagging changes like this without resorting to widespread file duplication, abuse of OOP, or an ad-hoc versioning system. It comes as no surprise that nobody in my organization was willing to do this sort of work.
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts