Testing on production • Hacker News Board

Re: Testing on production

#21

Post by sidlls » Mon, Aug 14, 2023, 11:38 AM UTC

Feature flags are just code, like the rest of the software. You can program any feature with it, including auto-enabling it given appropriate circumstances (e.g., the user is logged in to a developer account). Of course, that doesn't work for features available without requiring an account.

Re: Testing on production

#22

Post by mwint » Mon, Aug 14, 2023, 11:40 AM UTC

Feature flags sound great, but a company I’ve been consulting for has been using them to their own detriment. Seems like many bugs are due to a (production!) user not having the right combinations of flags enabled.

There ends up being code to deal with what happens when various combinations of flags are on/off, and that code doesn’t get tested much.

And teams spend a lot of time just removing flags.

This isn’t a safety-critical app - I really think they’d do better dropping the flags, and just deploying what they want when it’s ready.

Re: Testing on production

#23

Post by MH15 » Mon, Aug 14, 2023, 11:40 AM UTC

Yes, this is a relatively common practice. There’s of course still the chance you make a mistake setting up the feature flag and bring down production/expose the feature to users who shouldn’t have access.

Re: Testing on production

#24

Post by miguelxt » Mon, Aug 14, 2023, 11:42 AM UTC

I guess you can implement them however works for you and your team. I have personally implemented them in various way: depending on the date, on the client IP address, on an env var, on a logged user id, etc etc.

Re: Testing on production

#25

Post by devjab » Mon, Aug 14, 2023, 11:43 AM UTC

Yes. That’s how we typically do it in our shop. Though we do test it during development. Then when we think it’s ready, we have the product owner (or whoever ordered it) “play around with it” on a test setup. Before we let select users “test it in production”.

I’m not a fan of this article in general, however, a lot of what it talks about is anti-pattern in my book. Take the bit about Micro-services as an example. They are excellent in small teams, even when you only have 2-5 developers. The author isn’t wrong as such, it’s just that the author seems to misunderstand why Conway’s law points toward service architectures. Because even when you have 2-5 developers, the teams that actually ”own” the various things you build in your organisation might make up hundreds of people. In which case you’re still going to avoid a lot of complexity by using service architecture even if your developers sort of work on the same things.

Re: Testing on production

#26

Post by eloisius » Mon, Aug 14, 2023, 11:46 AM UTC

You’re describing QC. The reason that’s not sufficient is because your test user might not meet the conditions that trigger a bug. Trivial example: a bug that only shows up for users using RTL languages. A test suite allows you to test edge cases like that. Another shortfall of QC is that it doesn’t provide future assurance. A test suite makes sure the feature keeps working in the future when changes that interact with it are introduced.

Re: Testing on production

#27

Post by xnorswap » Mon, Aug 14, 2023, 11:46 AM UTC

I'm going to go further and say that Feature Flags are a nightmare and should be avoided. Because instead of just being used to stage roll-out, they get used to configure different environments for different customers.

You not only waste time with "Remove feature flag X" stories if all customers end up with the feature, you also slow down the response time of some categories of bugs, because you end up having to stop and check the combination of feature flags to reproduce a bug.

And if you end up with a feature that isn't popular except by one customer, not only are you now stuck supporting "Legacy feature Y", you're actually stuck supporting, "Optional legacy feature Y" which is worse.

Maybe I'm ranting about "misuse of feature flags", but I don't like to pontificate about how things ought to be, but how in my experience they actually are.

Re: Testing on production

#28

Post by rahoulb » Mon, Aug 14, 2023, 11:46 AM UTC

Yes.

As long as your code makes sure it takes account of that flag everywhere that it is used. Otherwise your new feature could "leak" into the system for everyone else.

Plus, as systems grow in complexity, there's always a danger that features step on each other. We'd like to think that everything we write is nicely isolated and separated from the rest of the system, but it never works that way - plus we're just a group of squishy humans who make mistakes. There will be times when having Features A and C switched on, with B switched off, produces some weird interactions that don't happen if A, B and C are switched on together.

Re: Testing on production

#29

Post by Narann » Mon, Aug 14, 2023, 11:49 AM UTC

> something that is hard but not impossible to go wrong may carry huge amounts of risk.

I think it's the definition of the black swan theory[1].

[1]: https://en.wikipedia.org/wiki/Black_swan_theory

Re: Testing on production

#30

Post by literallyroy » Mon, Aug 14, 2023, 11:49 AM UTC

Yes, feature flags are often able to be applied globally, or per customer. However, feature flags add complexity (littering your business logic with feature flag checks), so many small non-feature changes wouldn’t use them.