Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feature Flags: Theory vs. Reality

bpapillon.com

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Feature Flags: Theory vs. Reality

#51

Post by lucas_membrane »

I worked on a team of hundreds that developed and maintained a vertical market app enterprise app for a few thousand client companies, probably more than 100,000 end-user seats, but fewer than 500,000. My small sample size (1) observation is that the developer organizations least able to manage feature flags are the ones most likely to buy into such a magic pill cargo cult solution.

If your software has accumulated or is built to support numerous independent client organizations, it almost certainly has features that are not used by all users, and thereby the software has implicit feature flags embedded in the data that it is already processing. Regardless of whether those feature flags in data work well or work poorly, why in the world would you want to add a second feature-control subsystem? Because it is meta-programming, I suppose, and we all know that meta-programming just adds another level of power to everything, and your first feature-control system may be a little hard to disentangle, and you can make feature-flags work by having the meta-programming done by a select few who really know what they are doing, and it will be a worthwhile challenge, and even if it doesn't work you will learn a lot, and it will look good on your resume, and give everyone a few good laughs when they realize what they were trying to do.

Re: Feature Flags: Theory vs. Reality

#52

Post by withinboredom »

Probably my best story of “zombie flags” was when this guy accidentally deleted a production table. We disabled the feature flag, disabled some code written after it had been turned on and expected it to be on, then restored the table from a PIT backup. Finally, we reverted the code changes and feature flag. We were back up in a matter of hours (the table was hundreds of gb, so it took awhile to delete and restore). Some customers noticed the option missing from their options screen, but 99% of the customers never noticed the feature downtime.

Re: Feature Flags: Theory vs. Reality

#53

Post by mandelbrotwurst »

esafak wrote: post #42
adamesque wrote: post #38

Earlier quoted context omitted.

It’s actually usually the paying customers asking via support to be added to the holdback, improved experience or no. This is more true for larger flags that substantially change the experience and may not implement niche or edge-case functionality. Obviously you want to avoid these kinds of tests if possible but it’s not always possible.

Users should not be allowed to select their treatments; it defeats randomization, which is what allows causal inference.

Sure, they'll be more predictive that way, and simultaneously it's valuable to not piss off your customers.

Re: Feature Flags: Theory vs. Reality

#54

Post by deathanatos »

no_wizard wrote: post #41
zero_shift wrote: post #4

I've definitely lived with the zombie flags problem. Teams ship experiments that double the size of a piece of code, but never go back to refactor out the unused code branches. In shared codebases this becomes a nightmare of thousands of lines of zombie code and unit tests. This is a social problem as much as a technical one: even if you have LaunchDarkly, DataDog etc making very clear that a flag isn't used, getting…

This is the only way, more or less, to enforce any code standards, whether its refactoring, quality, testing, docs etc. If it doesn't break the build (or do anything else that stops it from moving forward) there will always be external pressure to get things out "ASAP" despite in most circumstances "ASAP" isn't required. If you can't full on stop whats happening, it becomes exponentially harder to enforce anything.

I agree with you, entirely, but I've had developers fight tooth and nail when the build breaks. "We need to get out there ASAP!" is definitely the cry, and usually "compromises" are made, such as "what if we made the overall CI run not fail if this test fails?" — which is as good as killing the test, IMO.

The impetus is necessary, or the problem is ignored.

The devs are really just proxies for the stress a PM is inappropriately pushing, though. But they are paid to not understand this problem, so getting them on board is impossible.

Re: Feature Flags: Theory vs. Reality

#55

Post by esafak »

mandelbrotwurst wrote: post #53
esafak wrote: post #42

Earlier quoted context omitted.

Users should not be allowed to select their treatments; it defeats randomization, which is what allows causal inference.

Sure, they'll be more predictive that way, and simultaneously it's valuable to not piss off your customers.

In that case I would take them out of the experiment and impute the censored data.

https://en.wikipedia.org/wiki/Censoring_(statistics)

Re: Feature Flags: Theory vs. Reality

#56

Post by marcosdumay »

6D794163636F756 wrote: post #35
marcosdumay wrote: post #21

Earlier quoted context omitted.

> getting a team to prioritise cleanup is difficult This is the limitation that breaks every development practice people come up with. That idea of formalizing the cleanup and requiring it for deployment is very interesting. It may be possible to extend it to other contexts.

It runs counter to the pressures a developer faces. I've at times been told that tech debt is fine because products only last 3-4 years before a replacement gets made. When you're on that timeline who cares if you've cleaned up after yourself?

> products only last 3-4 years before a replacement gets made

Is that your experience?

I imagine it can be a kind of self fulfilling prophecy, but even then I can't imagine people replacing everything each 4 years. And if that's not your experience, then the point is moot.

Re: Feature Flags: Theory vs. Reality

#57

Post by 6D794163636F756 »

marcosdumay wrote: post #56
6D794163636F756 wrote: post #35

Earlier quoted context omitted.

It runs counter to the pressures a developer faces. I've at times been told that tech debt is fine because products only last 3-4 years before a replacement gets made. When you're on that timeline who cares if you've cleaned up after yourself?

> products only last 3-4 years before a replacement gets made Is that your experience? I imagine it can be a kind of self fulfilling prophecy, but even then I can't imagine people replacing everything each 4 years. And if that's not your experience, then the point is moot.

That's how long it takes before the person who signed off on it has moved to another job and made the product someone else's problem. At that point the new person will usually kick off a new project because the old one is bad and releasing a new product is better for their chances of promotion

Re: Feature Flags: Theory vs. Reality

#58

Post by DenisM »

Couple of simple ideas for the zombie flag problem:

- When adding a flag immediately file a bug to remove the flag by a certain date. Enforce in code review. The bug count will surface the problem to the management.

- When a flag is past due date start firing non-fatal incidents. The incident count will also surface the problem to the management.

Re: Feature Flags: Theory vs. Reality

#59

Post by phyrex »

malfist wrote: post #19

One thing I see missing in this article is another huge cost to these things. What happens when your homegrown feature flag microservice (because why pay for a hard cost when you can have the soft cost of making your own) goes down, even temporarily. Sane defaults at code review time, before launch aren't always the sane defaults after a feature has fully launched, or nearly fully launched. I've seen more than a few…

At my job feature flags (and other configuration) get distributed as static files that replaced by config updates, so if there’s ever a disruption the hosts still have the last valid code configuration values

Re: Feature Flags: Theory vs. Reality

#60

Post by alexjurkiewicz »

zero_shift wrote: post #4

I've definitely lived with the zombie flags problem. Teams ship experiments that double the size of a piece of code, but never go back to refactor out the unused code branches. In shared codebases this becomes a nightmare of thousands of lines of zombie code and unit tests. This is a social problem as much as a technical one: even if you have LaunchDarkly, DataDog etc making very clear that a flag isn't used, getting…

What prevents teams extending the expiry date repeatedly?
Post reply on HN