Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Principles for building and scaling feature flag systems

docs.getunleash.io

Post reply on HN
Print view View original

101–110 of 115 posts

Re: Principles for building and scaling feature flag systems

#101

Post by gastonfournier »

baq wrote: post #31
mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

Or... see them for what they are: runtime configuration. The name implies a use case scenario, but in reality it's just a configuration knob. With a good UI, it's a pretty damn convenient way to do runtime configuration. So of course they'll be used for long-term configuration purposes, especially under pressure and for gradual rollouts of whole systems, not just A/B testing features.

The main challenge is when things go wrong. Feature flags are designed for high-rate evaluation with low latency responses. Configuration usually doesn't care that much about latency as it's usually read once at startup. This context leads to some very specific tradeoffs such as erring to availability over consistency, which in the case of configuration management could be a bad choice

Re: Principles for building and scaling feature flag systems

#102

Post by gastonfournier »

tantalor wrote: post #98
gastonfournier wrote: post #96

Earlier quoted context omitted.

It can be done by opening a PR, I haven't tried it yet, but I'm curious to try out https://github.com/uber/piranha or maybe hear some experiences if someone has used it

Which would be instantly rejected because the flag is still being used.

AFAIK, it'd only open a PR if the flag is fully enabled and has some heuristics to determine when it's safe to remove. Honestly, I haven't tested it but I'm curious to know if someone had either good or bad experiences.

If all the PRs are instantly rejected, that would be a bad sign, but I couldn't find someone who effectively used it. I mean, it's been around for a while but it didn't spread out, so that already gives me some hint

Re: Principles for building and scaling feature flag systems

#103

Post by gastonfournier »

tvink wrote: post #85

>Organizations who adopt feature flags see improvements in all key operational metrics for DevOps: Lead time to changes, mean-time-to-recovery, deployment frequency, and change failure rate. Is this true? unfortunately there's no sources indicated, and a quick check on scholar doesn't show me anything of the sort.

There are a few case studies listed in most of the feature flag solutions, of course, each organization is completely different and the maturity of each organization varies. But feature flags are a 2-way-door decision, meaning that you can adopt them at smaller scale, try it out and see if it works for you before making a decision.

Here's a list of case studies from some of the solutions referred in the comments, some focus on operational metrics, others in lead time to changes: https://www.getunleash.io/case-studies https://launchdarkly.com/case-studies/ https://www.flagsmith.com/case-studies

Re: Principles for building and scaling feature flag systems

#104

Post by timothyfcook »

strken wrote: post #73
mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

I think it's better to admit they actually are config, just a different kind of config that comes with an expiration date. Accepting reality in this way means you'll design a config management system that lets you add feature flags with a required expiration date, and then notifies you when they're still in the system after the deadline.

I agreed. My perspective is that there are two kinds of feature flags: temporary and permanent.

Temporary ones can be used to power experiments or just help you get to GA and then can be removed.

Permanent ones can be configs that serve multiple variations (e.g. values for rate limits), but they can also be simple booleans that manage long term entitlements for customers (like pricing tiers, regional product settings, etc.)

Re: Principles for building and scaling feature flag systems

#105

Post by timothyfcook »

tvink wrote: post #85

>Organizations who adopt feature flags see improvements in all key operational metrics for DevOps: Lead time to changes, mean-time-to-recovery, deployment frequency, and change failure rate. Is this true? unfortunately there's no sources indicated, and a quick check on scholar doesn't show me anything of the sort.

The LaunchDarkly 2022 State of Feature Management Report has results from surveying 1000 software people and looks at impact on those DevOps metrics: https://launchdarkly.com/state-of-feature-management/

Re: Principles for building and scaling feature flag systems

#106

Post by tantalor »

gastonfournier wrote: post #102
tantalor wrote: post #98

Earlier quoted context omitted.

Which would be instantly rejected because the flag is still being used.

AFAIK, it'd only open a PR if the flag is fully enabled and has some heuristics to determine when it's safe to remove. Honestly, I haven't tested it but I'm curious to know if someone had either good or bad experiences. If all the PRs are instantly rejected, that would be a bad sign, but I couldn't find someone who effectively used it. I mean, it's been around for a while but it didn't spread out, so that already giv…

If the cleanup only happens if the flag is not used, then the "expiration date" is basically meaningless. You can either delete it or you can't. Who cares if it's expired or not.

Re: Principles for building and scaling feature flag systems

#107

Post by zellyn »

baq wrote: post #68
zellyn wrote: post #9

Earlier quoted context omitted.

Oh, and one last(?) update. If you create your own service to evaluate a bunch of feature flags for a given user/client/device/location/whatever and return the results, for use in mobile clients (everyone does this), PLEASE *make sure the client enumerates the list of flags it wants*. It's very tempting to just keep that list server-side, and send all the flags (much simpler requests, right?), but you will have to ke…

You should be collecting metrics on used flags and their values if you’re rolling your own. A saas offering will do that for you.

Well, it seems to be a common theme to build a server that uses the flag eval _server_ SDK to evaluate a bunch of flags and then pass them back to the client.

For example, a client may call myserver.com/mobile-flags?merchant=abcdef&device=123456&os=ios&os_version=15.2&app_version=6.1 and the server will pass back: flag1: true flag2: 39 flag3: false flag4: green

This seems to be a common theme. For example, LaunchDarkly has a mobile client SDK, but they charge by MAU, which would be untenable. So folks tend to write a proxy for the mobile apps to call. If the client (as in my example above) doesn't specify which flags it wants, then the metrics are missing, whether you're using a commercial product or your own: it'll simply tell you that all the flags got used. (Of course, you could be collecting metrics from the client apps).

But based on our experience, you'd be better of having the mobile client pass in an explicit list of desired flags. Which will give accurate metrics.

Hope that clarifies what I meant.

Re: Principles for building and scaling feature flag systems

#108

Post by zellyn »

kiitos wrote: post #94
zellyn wrote: post #8

Earlier quoted context omitted.

One more update. I spent a little time the other day trying to find all the feature flag products I could. I'm sure I missed a ton. Let me know in the comments! LaunchDarkly Split Apptimize CloudBees ConfigCat DevCycle FeatBit FeatureHub Flagsmith Flipper Flipt GrowthBook Harness Molasses OpenFeature Posthog Rollout Unleash Here's my first draft of the questions you'd want to ask about any given solution: Questionnai…

> Are flags evaluated in-memory, locally? (Hopefully!) This seems like a MUST rather than a SHOULD, right?

I would have thought so. But flagsmith apparently does primarily server-side eval. And even OpenFeature has `flagd`, which I guess is a sidecar, so a sort of hybrid approach.

And LaunchDarkly's Big Segments fetch segment inclusion data live from redis (although I believe they then cache it for a while).

Re: Principles for building and scaling feature flag systems

#109

Post by sporkland »

zellyn wrote: post #25
mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

Sadly, this is a battle you are destined to lose. I have almost completely given up. The best you can aim for is to use feature flags better rather than worse. - Some flags are going to stay forever: kill switches, load shedding, etc. (vendors are starting to incorporate this in the UI) - Unless you have a very-easy-to-use way to add arbitrary boolean feature toggles to individual user accounts (which can become its…

We have an interesting hybrid between the two that I'd like your take on. When we release new versions of our web client static assets we have a version number that we bump that moves folks over to the new version.

1. We could stick it in a standard conf system and serve it up randomly based on what host a client hits. (Or come up with more sophisticated rollouts)

2. Or we can put it as "perm" conf in the feature flag system and roll it out based on different cohorts/segments.

I'm leaning towards #2 but I'd love to understand why you want to prohibit long lived keys so I can make a more informed choice. The original blog posts main reasons were that FF systems favor availability over consistency so make a pour tool if you need fast converging global config, which somewhat becomes challenging here during rollbacks but is likely not the end of the world.

Re: Principles for building and scaling feature flag systems

#110

Post by bullcitydev »

oddx wrote: post #100

I dedicated a day to evaluating feature flag software based on specific criteria: - Must support multiple SDKs, including Java and Ruby. - Should be self-hosted with PostgreSQL database support. - Needs to enable remote configuration for arbitrary values (not just feature flags). I don't run two separate services for this. - Should offer some UI functionality. - it should cache flag values locally and, ideally, provi…

Hey thanks for giving Flipt a look! I'm the creator of Flipt so would love to chat more about your needs to see how we could make it work for your use case! We're actively looking into providing local caching for all our SDKs btw and would love to learn more about what your requirements are for remote configuration as it's also on our radar! Feel free to send me an email at: mark (at) flipt.io.
Post reply on HN