Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing on production

marcochiappetta.medium.com

Post reply on HN
Print view View original

61–70 of 101 posts

Re: Testing on production

#61

Post by marcosdumay »

perrygeo wrote: post #60
civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful. But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single appl…

The solution is to remove your feature flags after you are done with them.

Re: Testing on production

#62

Post by Eddygandr »

hef19898 wrote: post #38
contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

Or, one cpuld alao just follow some standard processes all the time , instead of developing an individual approach every single time. Those standards should contain mitigation for most of the common risks, and rules to apply for the rest. And one of those standards, an no I don't give a shit about developer experience, software or otherwise, should be you never ever test on production . As soon as you work on real pr…

There’s many reasons you test in production! The most recent TiP project I worked on was running a regression suite against production to notify outages before before users did (in an irregularly used but critical system, such that you can’t just go by user logs alone).

Re: Testing on production

#63

Post by [deleted] »

chpmrc wrote: post #46

Welp! For some reason someone at HN decided to change the title and bump this down to the 11th position (atm). Not sure what I did wrong here but it feels pretty crappy... @dang any chance you could help here? :(

[deleted]

Re: Testing on production

#64

Post by Eddygandr »

marcosdumay wrote: post #61
perrygeo wrote: post #60

Earlier quoted context omitted.

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful. But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single appl…

The solution is to remove your feature flags after you are done with them.

The problem is when you use feature flags for customer-bespoke reasons or to enable paid features. Then they’re always there and have to be tested in combinations which sucks.

Re: Testing on production

#66

Post by marcosdumay »

Eddygandr wrote: post #64
marcosdumay wrote: post #61

Earlier quoted context omitted.

The solution is to remove your feature flags after you are done with them.

The problem is when you use feature flags for customer-bespoke reasons or to enable paid features. Then they’re always there and have to be tested in combinations which sucks .

Yeah, those things are called "user settings". If you need them, you need them, but pretending they are feature flags and trying to port the flags development methods into your settings will lead to nothing but tears.

Re: Testing on production

#67

Post by sandworm101 »

contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

>> Something is only an actual risk if it hurts the bottom-line.

Sometimes not even that. We have see many huge breakdowns in recent years that did hit bottom lines, but didn't impact stock prices. Perhaps, at least for a publicly-traded company, the only real risks are those that might impact stock prices. That might include things that hurt other companies if doing so might result in money leaving the entire sector.

Re: Testing on production

#68

Post by seanthemon »

Eddygandr wrote: post #64
marcosdumay wrote: post #61

Earlier quoted context omitted.

The solution is to remove your feature flags after you are done with them.

The problem is when you use feature flags for customer-bespoke reasons or to enable paid features. Then they’re always there and have to be tested in combinations which sucks .

It's so closely related and so often mistaken, feature flags and tenant features are two completely different things

Re: Testing on production

#69

Post by kawemi »

contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

It's a really useful perspective in real-life scenarios when you're not developping critical software. Of course a baseline of risk-avoidance is always important, but businesses/custommers/users most of the time are ready to handle some risks, like downtime, bugs, delays, etc. SWE and developpers are the more risk-averse of the two parties, which leads to us over-valuing the importance of robustness and stability.

For example, it's way easier/faster to implement observability and some sort of rollback of bad versions than to try and prevent every possible way an app could crash and trigger a bunch of problems. What's going to happen if the app crash is pretty simple : customers will be mad (CS/Marketing/PR can handle them), you'll notice the downtime quickly and rollback (or maybe even rollback automatically!). Then you'll be in a perfect position to handle what went wrong : systems will be back on a known stable position and all the stress of trying to fix something in a live production system will be gone.

Re: Testing on production

#70

Post by andreareina »

perrygeo wrote: post #60
civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful. But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single appl…

2^nflags actually. Which is a much bigger number.
Post reply on HN