Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing on production

marcochiappetta.medium.com

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Testing on production

#71

Post by angarg12 »

contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

> may in fact be doing harm without actually sufficiently preventing any risk.

The biggest practical impediment to increasing velocity of delivery that I encounter is trying to convey this. People can visualize and estimate the risk and impact of a deployment gone wrong, but have a hard time estimating the impact of processes that slow down delivery. Therefore they overindex in heavy and "safe" processes (which often don't increase safety) at the cost of speed of iteration.

I'm not sure how to define this asymmetry, maybe some variation of loss aversion.

Re: Testing on production

#72

Post by chpmrc »

Shrezzing wrote: post #54
solatic wrote: post #44

Earlier quoted context omitted.

It's implied that it isn't easy to distinguish them during interviews . After they join your team, it's very easy to distinguish them.

I've re-read the developer experience section, and I can't see where that implication is established. In that context, the paragraph stands out as an abrupt diversion from the main theme of the section, and undermines the argument of the entire piece. The section defines developer dissonance, and asserts that it's possible to overcome it with reasoned and sensible questioning. If it's possible to overcome dissonance…

I can't see how making that statement undermines the rest of the argument. It would help if you could clarify that relationship.

And I'm not sure I understand why, if you can't distinguish them, the distinction is unimportant. It's hard to distinguish an edible mushroom from a poisonous one and yet making that distinction makes a huge difference.

Interviews are definitely a limited tool to do so btw, this is only something that you realize over time. It's also very easy to play an interviewer if the interviewee's soft skills are better than the interviewer's (which happens often in this industry).

Re: Testing on production

#73

Post by chpmrc »

chpmrc wrote: post #46

Welp! For some reason someone at HN decided to change the title and bump this down to the 11th position (atm). Not sure what I did wrong here but it feels pretty crappy... @dang any chance you could help here? :(

For the sake of transparency it was explained to me that the title was too "link baity" and that the comment section was a bit too heated. I appreciate the explanation and I agree this kind of moderation is, unfortunately, required to keep things civil and constructive.

Re: Testing on production

#74

Post by al_be_back »

>> If Tesla makes a mistake in their autopilot software, people might die.

In this case, a good "Testing on Production" rule would be to not let customers test your software, period.

There's plenty of land and resources to construct towns and cities that simulate real-life commute very accurately.

In the case of self-driving (or even autopilot), you're not really testing a feature, you're researching a new product, they difference is vast.

Re: Testing on production

#76

Post by radlad »

kawemi wrote: post #69
contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

It's a really useful perspective in real-life scenarios when you're not developping critical software. Of course a baseline of risk-avoidance is always important, but businesses/custommers/users most of the time are ready to handle some risks, like downtime, bugs, delays, etc. SWE and developpers are the more risk-averse of the two parties, which leads to us over-valuing the importance of robustness and stability. Fo…

Of course, there is no magic bullet. Some problems aren't solved by rolling back services. (e.g. A thundering herd of clients caused by re-deploying an old build overloading your database.)

Re: Testing on production

#77

Post by recroad »

perrygeo wrote: post #60
civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful. But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single appl…

Long lived feature flags are totally fine, they're more like operational flags than anything. The Fowler article is pretty good at classifying them. Depending on the type of flag (longevity/dynamism) the design will vary. https://martinfowler.com/articles/feature-toggles.html

Re: Testing on production

#78

Post by kiitos »

recroad wrote: post #77
perrygeo wrote: post #60

Earlier quoted context omitted.

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful. But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single appl…

Long lived feature flags are totally fine, they're more like operational flags than anything. The Fowler article is pretty good at classifying them. Depending on the type of flag (longevity/dynamism) the design will vary. https://martinfowler.com/articles/feature-toggles.html

An essential property of a feature flag is that it is short-lived, existing only for the duration of the roll-out of the feature. In the language of your linked article, feature flags are 1-to-1 with "release toggles" and not really any other kind of toggle.

Re: Testing on production

#79

Post by hulitu »

> Shipping confidence We can define “shipping confidence” as the feeling a mentally sane developer has when they know their code is about to be deployed to production (whether it can be updated over the air or not).

A bug which must be fixed in production is much more expensive than a bug fixed during development.

People here complain when you bash Microsoft, but their phylosophy was (and still is) let the users test the product.

Re: Testing on production

#80

Post by kiitos »

Eddygandr wrote: post #64
marcosdumay wrote: post #61

Earlier quoted context omitted.

The solution is to remove your feature flags after you are done with them.

The problem is when you use feature flags for customer-bespoke reasons or to enable paid features. Then they’re always there and have to be tested in combinations which sucks .

Echoing sibling comments, feature flags are about managing the deployment of new product capabilities, and should always be short-lived. They're not an appropriate choice for any kind of long-lived capability, like anything that's per-customer, or paid vs. non-paid, or etc. Using feature flags for those kinds of things is a classic design mistake.
Post reply on HN