Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing on production

marcochiappetta.medium.com

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Testing on production

#51

Post by JohnBooty »

ethbr1 wrote: post #12
trollied wrote: post #3

"Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in"

I code at the interface between ops teams (on the business side) of companies and dev teams (on the IT side). One of the things I've realized is that in most unregulated companies (read: non-healthcare/financial) the business side of the house is used to having little or no lower lifecycle. If they want to make a process change, they make it on production work. Granted, they have change control approvals, etc. etc.,…

This hasn't been my experience. I think it depends on how business-critical the application is.

I worked at a home remodeling company. Revenue was several million dollars a day. App handled sales, scheduling, logistics, everything. Breaking production was a big deal, it cost us millions per day and created logjams.

I would think that most online applications are the same. Even if a simple online web shop goes down you are costing money.

What kinds of experiences have you had where testing in production was the norm?

    because you can't do certain things without lower environments.
I agree that this is something many shops REALLY struggle with.

One of the most challenging things is exporting or creating some kind of realistic data set for local development use. I think 99% of companies struggle with this.

Re: Testing on production

#52

Post by contravariant »

HL33tibCe7 wrote: post #34
contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

Risk = likelihood * severity

That's also an approach, but it may lead to endless discussions about how likely something is. It's easier to tell what the worst possible consequence is (what this article calls criticality). After that it's fairly straightforward to figure out if you're doing enough to prevent this scenario from being realized (which is more like coverage = risk * mitigations).

Re: Testing on production

#53

Post by datadrivenangel »

jiggawatts wrote: post #35
civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

This is pretty common at larger scales, and is also often done on a per-tenant or per-account basis. For example, the Microsoft Azure public cloud has a hierarchy of tenant -> subscription -> resource group -> resource. It's possible to have feature flags at all four levels, but the most common one I see is rolling deployments where they pick customer subscriptions at random, and deploy to those in batches. This mean…

The training aspect of feature flags is a huge pain point.

Not to mention it looks really awkward when an account manager has forgotten to enable some great new feature for you.

Re: Testing on production

#54

Post by Shrezzing »

solatic wrote: post #44
Shrezzing wrote: post #42

I enjoyed the entire article except this part: > Unfortunately there is no easy way to distinguish between people who are good and need a paycheck from people who just need a paycheck. But you sure as hell don’t want the latter in your team. If you can't tell them apart, then the distinction is unimportant. So if among the group of people who need paychecks, good is indistinguishable from non-good, the comment serves…

It's implied that it isn't easy to distinguish them during interviews . After they join your team, it's very easy to distinguish them.

I've re-read the developer experience section, and I can't see where that implication is established. In that context, the paragraph stands out as an abrupt diversion from the main theme of the section, and undermines the argument of the entire piece. The section defines developer dissonance, and asserts that it's possible to overcome it with reasoned and sensible questioning. If it's possible to overcome dissonance with reasoned questioning, a hiring interview should a prime opportunity to roll out some reasoned questions and head-off dissonance before it enters the organisation in the first place.

Re: Testing on production

#55

Post by whstl »

JohnBooty wrote: post #45
xnorswap wrote: post #27

Earlier quoted context omitted.

I'm going to go further and say that Feature Flags are a nightmare and should be avoided. Because instead of just being used to stage roll-out, they get used to configure different environments for different customers. You not only waste time with "Remove feature flag X" stories if all customers end up with the feature, you also slow down the response time of some categories of bugs, because you end up having to stop…

They can be very very very nice if you have a lengthy (or perhaps just unpredictable) build/deploy process. And/or if you have lots of teams working independently on the same monolith. Suppose you have daily production builds. You are rolling out Feature XYZ. You would like to enable it in prod, but you would like to monitor it closely and may need to turn it off again. Feature flags allow that. Ultimately what's bei…

I also use feature flags when I'm 100% sure stakeholders or PMs will somehow find fault a certain feature after it's deployed, even though they're the ones who specified it, approved it and tested it in a staging environment.

Not exactly the thing that we should be using Feature Flags for, but it saved my ass several times.

On the other hand: this removes some of the accountability that non-technical folks have over software. This can be detrimental in the long term.

Re: Testing on production

#56

Post by kasey_junk »

hef19898 wrote: post #38
contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

Or, one cpuld alao just follow some standard processes all the time , instead of developing an individual approach every single time. Those standards should contain mitigation for most of the common risks, and rules to apply for the rest. And one of those standards, an no I don't give a shit about developer experience, software or otherwise, should be you never ever test on production . As soon as you work on real pr…

Every enterprise I’ve ever worked for, including corporate giants running the backbones of modern finance ran tests on production. Post deploy smoke tests, fail over tests, small group alphas, test migration/roll backs etc.

Not being confident in your test plan is a sign of immaturity not maturity because at some point you are going to need to validate how something behaves in production.

There are a wide range of processes, procedures and software architectures to get you to being confident that your production testing is doing more good than harm for your customers but in an environment where you can deploy new software you are going to do some testing in production.

Re: Testing on production

#57

Post by rco8786 »

civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

Yes, that's the general idea - and it works pretty well.

It can also be a huge PITA. The fallacy is that a "feature" is an isolated chunk of code. You just wrap that in a thing that says "if feature is on, do the code!". But in reality, a single feature often touches numerous different code points, potentially across multiple codebases and services/APIs. So you have to intertwine that feature flag all over the place. Then write tests that test for each scenario (do the right thing when the feature is off, do the right thing then the feature is on). Then you have to remember to go back and clean up all that code when the feature is on for everyone and stabilized.

It's a good tool, but it's not an easy tool like a lot of folks think it is.

Re: Testing on production

#58

Post by bradleyjg »

civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

A note of caution re: flags from an oracle dev: https://news.ycombinator.com/item?id=18442941

Re: Testing on production

#59

Post by hef19898 »

datadrivenangel wrote: post #50
hef19898 wrote: post #39

Earlier quoted context omitted.

Sometimes one has to include detectability as well.

Severity should include detectability. If you never detect an issue, it's not an issue because nobody sees it.

Usually it is a seperate factor, at least as far as P/D-MEAs are concerned. Quick and dirty, sure, it can be included in severity. Personally, I prefer the increased transparency and granularity of having detectability as a different factor.

Re: Testing on production

#60

Post by perrygeo »

civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful.

But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single application, you have n_flags ^ 2 applications that all behave in subtlety different ways depending on the interaction of the flags.

There's no way around it - you have to test all branches of your code somehow. "Just let the users find the bugs" doesn't work in this case since each user can only test their unique combination of flags. I've regularly seen default and QA tester flag configurations work great, only to have a particular combination fail for customers.

The only solution is setting up a full integration test for every combination of flags. If that sounds tedious (and it is), the solution is to avoid feature flags, not to avoid testing them!

Post reply on HN