Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Principles for building and scaling feature flag systems

docs.getunleash.io

Post reply on HN
Print view View original

21–30 of 115 posts

Re: Principles for building and scaling feature flag systems

#21

Post by zellyn »

daigoba66 wrote: post #11
zellyn wrote: post #6

Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause product…

> One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant). Can you elaborate on this? As a programmer, I would think that using something like a constant would help us find references and ensure all usage of the flag is removed when the constant is removed.

One of the most common things you want to do for a feature flag or metric name is ask, "Where is this used in code?". (LaunchDarkly even has a product feature that does this, called "Code References".) I suppose one layer of indirection (into a constant) doesn't hurt too much, although it certainly makes things a little trickier.

The bigger problem is when the code constructs metric and flag names programmatically:

    prefix = "framework.client.requests.http.{status%100}s"
    recordHistogram(prefix + ".latency", latency)
    recordCount(prefix + ".count", 1)

    flagName = appName + "/loadshed-percent"

    # etc...
That kind of thing makes it very hard to find references to metrics or flags. Sometimes it's impossible, or close to impossible to remove, but it's worth trying hard.

Of course, this is just, like, my opinion, man!

Re: Principles for building and scaling feature flag systems

#22

Post by snorlaxmorlax »

zellyn wrote: post #7
zellyn wrote: post #6

Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause product…

After typing that, and realizing I have a lot more to say, I guess I should write a blog post on the subject

You definitely should! These questions are great, and could use some appropriate context for evaluation.

Re: Principles for building and scaling feature flag systems

#23

Post by zellyn »

vlovich123 wrote: post #13
zellyn wrote: post #8

Earlier quoted context omitted.

One more update. I spent a little time the other day trying to find all the feature flag products I could. I'm sure I missed a ton. Let me know in the comments! LaunchDarkly Split Apptimize CloudBees ConfigCat DevCycle FeatBit FeatureHub Flagsmith Flipper Flipt GrowthBook Harness Molasses OpenFeature Posthog Rollout Unleash Here's my first draft of the questions you'd want to ask about any given solution: Questionnai…

Do you have the answers to that questionnaire for the services you mention?

I more or less know all the answers for LaunchDarkly (except pricing details), and for the internal feature flag service we're deprecating, but I haven't gone through and answered it for all the other offerings. It would be time-consuming, but very useful.

Also, undoubtedly contentious. If you want an amusing read, go check out LaunchDarkly's "comparison with Split" page and Split's "comparison with LaunchDarkly" page. It's especially funny when they make the exact same evaluations, but in reverse.

Re: Principles for building and scaling feature flag systems

#24

Post by zellyn »

zellyn wrote: post #6

Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause product…

> I'd argue that delivering flag definitions is (relatively) easy.

I'd argue that coming up with good UI that nudges developers towards safe behavior, as well as useful and appropriate guard rails -- in other words, using the feature flag UI to reduce likelihood of breakage -- is difficult, and one of the major value propositions of feature flag services.

Re: Principles for building and scaling feature flag systems

#25

Post by zellyn »

mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

Sadly, this is a battle you are destined to lose. I have almost completely given up. The best you can aim for is to use feature flags better rather than worse.

    - Some flags are going to stay forever: kill switches, load shedding, etc. (vendors are starting to incorporate this in the UI)
    - Unless you have a very-easy-to-use way to add arbitrary boolean feature toggles to individual user accounts (which can become its own mess), people are going to find it vastly easier to create feature flags with per-use override lists (almost all of them let you override on primary token). They will use your feature flags for:
      - Preview features: "is this user in the preview group?"
      - rollouts that might not ever go 100%: "should this organization use the old login flow?"
      - business-critical attributes that it would be a major incident to revert to defaults: "does this user operate under the alternate tax regime?"
You can try to fight this (indeed, especially for that last one, you most definitely should!), but you will not ever completely win the feature flag ideological purity war!

Re: Principles for building and scaling feature flag systems

#26

Post by ferrantim »

zellyn wrote: post #7
zellyn wrote: post #6

Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause product…

After typing that, and realizing I have a lot more to say, I guess I should write a blog post on the subject

Yes please. Blog would be awesome.

Re: Principles for building and scaling feature flag systems

#27

Post by gastonfournier »

mabbo wrote: post #12

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another d…

I faced something similar, and I think it's unavoidable. Give people a screwdriver and they'll find a way of using it as a hammer.

The best you can do is expect the feature flagging solution to give some kind of warning for tech debt. Then equip them with alternative tools for configuration management. Rather than forbidding, give them options, but if it's not your scope, I'd let them be (I know as engineers this is hard to do :P).

Re: Principles for building and scaling feature flag systems

#28

Post by gastonfournier »

tiberriver256 wrote: post #5

Item #1 depends on the reason you're using feature flags. For a more nuanced and careful discussion of the topic I like to reference: https://martinfowler.com/articles/feature-toggles.html

It's true that there are more long-lived use cases, but if you have the ability to choose, runtime controlled ones cover both cases, while compile time only cover some use cases. But fair point

Re: Principles for building and scaling feature flag systems

#29

Post by angarg12 »

Offtopic but relevant:

TL;DR if you break long posts into pages, at least have an option to see the whole thing in a single page.

I use a browser extension to send websites to my Kindle. It's great for long-ish format blog posts that I want to read, but I don't have the time at the moment. However, whenever I see long blog posts that are broken into sections, each one in it's own page, it becomes a mess. It forces me to navigate each individual page and send it to my Kindle. Then in the Kindle I have a long list of unsorted files that I need to jump around to read in order.

I understand breaking long pieces of text into pages makes it neater and more organized, but at least have an option to see the whole thing in a single page, as a way to export it somewhere else for easy reading.

Re: Principles for building and scaling feature flag systems

#30

Post by gastonfournier »

aranchelk wrote: post #14

With regard to web-based services, once you’ve got the ability to do canary testing, IMO flags/toggles are less compelling — busier code and logic you’ll have to pull out later.

I agree that if you have only a few changes going to prod, fast and doing canary testing, you should be covered. In my experience that's rarely the case because of multiple teams deploying changes at the same time, and even deployments in external services causing side effects in other services.
Post reply on HN