An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…
Testing on production
31–40 of 101 posts
Re: Testing on production
#32Earlier quoted context omitted.
Feature flags sound great, but a company I’ve been consulting for has been using them to their own detriment. Seems like many bugs are due to a (production!) user not having the right combinations of flags enabled. There ends up being code to deal with what happens when various combinations of flags are on/off, and that code doesn’t get tested much. And teams spend a lot of time just removing flags. This isn’t a safe…
I'm going to go further and say that Feature Flags are a nightmare and should be avoided. Because instead of just being used to stage roll-out, they get used to configure different environments for different customers. You not only waste time with "Remove feature flag X" stories if all customers end up with the feature, you also slow down the response time of some categories of bugs, because you end up having to stop…
Or you can have infinite permutations of feature flags if you don't flip them to be on for everyone quickly enough, and it becomes hard to test if[a&(not b)&c&(not d)] behavior vs if[a&b&(not c)&d] and.... you end up with too many to cover with testing well.
Re: Testing on production
#33I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…
Re: Testing on production
#34An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…
Re: Testing on production
#35I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…
For example, the Microsoft Azure public cloud has a hierarchy of tenant -> subscription -> resource group -> resource.
It's possible to have feature flags at all four levels, but the most common one I see is rolling deployments where they pick customer subscriptions at random, and deploy to those in batches.
This means you can have a scenario where your tenant (company) is only partially enabled for a feature, with some departments having subscriptions with the feature on, but others don't have it yet.
This can be both good and bad. The blast radius of a bad update is minimised, but the users affected don't care how many other users aren't affected! Similarly, inconsistencies like the one above are frustrating. Even simple things like demonstrating a feature for someone else can result in accidental gaslighting where you swear up and down that they just need to "click here" and they can't find the button...
Re: Testing on production
#36I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…
Classic story: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...
Re: Testing on production
#37I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…
Feature flags sound great, but a company I’ve been consulting for has been using them to their own detriment. Seems like many bugs are due to a (production!) user not having the right combinations of flags enabled. There ends up being code to deal with what happens when various combinations of flags are on/off, and that code doesn’t get tested much. And teams spend a lot of time just removing flags. This isn’t a safe…
Definitely doesn't do anything like completely obviate the issue though.
Re: Testing on production
#38An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…
And one of those standards, an no I don't give a shit about developer experience, software or otherwise, should be you never ever test on production. As soon as you work on real products for real customers you better start behaving like a professional. Childs play is over as soon as some one is paying you to do stuff.
Re: Testing on production
#39An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…
Risk = likelihood * severity
Re: Testing on production
#40Earlier quoted context omitted.
Feature flags sound great, but a company I’ve been consulting for has been using them to their own detriment. Seems like many bugs are due to a (production!) user not having the right combinations of flags enabled. There ends up being code to deal with what happens when various combinations of flags are on/off, and that code doesn’t get tested much. And teams spend a lot of time just removing flags. This isn’t a safe…
I'm going to go further and say that Feature Flags are a nightmare and should be avoided. Because instead of just being used to stage roll-out, they get used to configure different environments for different customers. You not only waste time with "Remove feature flag X" stories if all customers end up with the feature, you also slow down the response time of some categories of bugs, because you end up having to stop…
That is not a feature flag, that is a customer configuration option. They are different things and should not be treated in the same way.
Sure, it is possible for a feature flag to behave like a configuration option but they have different lifecycles and different audiences and so should not be confused. Of course it is easy to say that but harder in practice to maintain those differences.