Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing on production

marcochiappetta.medium.com

Post reply on HN
Print view View original

81–90 of 101 posts

Re: Testing on production

#81

Post by chpmrc »

JohnBooty wrote: post #47
chpmrc wrote: post #9

Earlier quoted context omitted.

Oh what an accomplishment it would be, to be able to change the meaning of the word "smart" with a single article! (Don't take it too seriously, like I said this is mostly a brain dump, I'm sure there's a lot of stuff that can be improved)

I like your usage of "smart" in the article. I see this challenge a lot in the industry. The young engineers truly are smart, even brilliant, but lack wisdom and experience.

I completely agree. "Smart" isn't used sarcastically here. It's an adjective that most young devs would (rightfully) like to be referred to as. But I see experienced devs as less interested in looking/being "smart" (or clever or whatever word you want to use) and just getting things done in a way that allows the org to make money and get rid of BS (unrelated to the above) as much as possible.

Maybe there's a better way to outline this difference.

Re: Testing on production

#83

Post by chias »

contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

This was an interesting bit of math I did when I joined a startup. It's pretty counter-intuitive to think about how very large numbers can increase the importance of small ones.

Say the company you work for is worth $10,000,000, and that you're hosted on GCP. Now take your best guess: what do you think the likelihood is of e.g. a fire or earthquake or something occurring in all relevant Google infrastructure simultaneously*, basically ushering in the end of all of your infrastructure, data, and backups? Frame that in a number of years. Is this kind of event something that may happen once in a thousand years? Once in ten thousand years? Let's say this is the sort of thing that might happen once in ten thousand years -- that's a long time!

Then the cost of this particular risk to your company is $1000 / year.

This kind of math isn't just a toy. When you have questions like "would maintaining actual physical backups in a safe somewhere outside of GCP be worth it?", you now have a framework to answer them ("if it would cost less than $1000 per year, then yes")

--

* or substitute in your favorite company-ending event.

Re: Testing on production

#84

Post by nonameiguess »

Shrezzing wrote: post #42

I enjoyed the entire article except this part: > Unfortunately there is no easy way to distinguish between people who are good and need a paycheck from people who just need a paycheck. But you sure as hell don’t want the latter in your team. If you can't tell them apart, then the distinction is unimportant. So if among the group of people who need paychecks, good is indistinguishable from non-good, the comment serves…

There is something I've started to notice as I've been working as a platform-layer consultant for the past few years. Many of the companies I've worked with don't have anyone in their company with any meaningful level of experience or expertise in environment administration, security, really anything ops-related. I see this especially when I start trying to hand off work I've done into maintenance phase operations and they don't have any kind of operations team to take over, but my contract sure as shit doesn't say I'm going to come in at 3 AM on a Sunday morning and I never will. So they may try to identify someone in the company to train up or they may try to hire, but the core problem they face isn't that it's impossible in principle to tell good apart from bad. The problem is it's impossible for them to tell the difference because they have no one in their company even qualified to conduct such an interview.

Re: Testing on production

#85

Post by kawemi »

radlad wrote: post #76
kawemi wrote: post #69

Earlier quoted context omitted.

It's a really useful perspective in real-life scenarios when you're not developping critical software. Of course a baseline of risk-avoidance is always important, but businesses/custommers/users most of the time are ready to handle some risks, like downtime, bugs, delays, etc. SWE and developpers are the more risk-averse of the two parties, which leads to us over-valuing the importance of robustness and stability. Fo…

Of course, there is no magic bullet. Some problems aren't solved by rolling back services. (e.g. A thundering herd of clients caused by re-deploying an old build overloading your database.)

Yes of course, my fake situation was assuming a pretty boring case of failure with an easy out (rollback). The underlying principle is that most of the time trying to preempt every situation is way more work than being conscious of them and giving yourself and your team(s) reasonable tools to mitigate them :)

Re: Testing on production

#86

Post by steveBK123 »

whstl wrote: post #55
JohnBooty wrote: post #45

Earlier quoted context omitted.

They can be very very very nice if you have a lengthy (or perhaps just unpredictable) build/deploy process. And/or if you have lots of teams working independently on the same monolith. Suppose you have daily production builds. You are rolling out Feature XYZ. You would like to enable it in prod, but you would like to monitor it closely and may need to turn it off again. Feature flags allow that. Ultimately what's bei…

I also use feature flags when I'm 100% sure stakeholders or PMs will somehow find fault a certain feature after it's deployed, even though they're the ones who specified it, approved it and tested it in a staging environment. Not exactly the thing that we should be using Feature Flags for, but it saved my ass several times. On the other hand: this removes some of the accountability that non-technical folks have over…

I have also found that for UIs the best thing to do is have a staged rollout approach.

Internals / Friendly users / Less friendly users / VIPs. The blast radius & intensity of explosion is smaller on the earlier groups.

The groups themselves need not be fixed. If you have a stakeholder/group that demanded the new features, they can be in an early wave. Inevitably they may be the ones to find defects in it, so the sooner the better.

Re: Testing on production

#87

Post by big_question »

chias wrote: post #83
contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

This was an interesting bit of math I did when I joined a startup. It's pretty counter-intuitive to think about how very large numbers can increase the importance of small ones. Say the company you work for is worth $10,000,000, and that you're hosted on GCP. Now take your best guess: what do you think the likelihood is of e.g. a fire or earthquake or something occurring in all relevant Google infrastructure simultan…

Similarly:

"We are spending 50$ per month just for one test in our code. We could cut it down to 10$ if we wanted."

"How many hours would it take to reduce the spend? If it's more than a couple of hours for a senior engineer, then it's not worth it."

We kept spending money on this inefficient test and it was the right choice.

Re: Testing on production

#88

Post by Scubabear68 »

contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

Very well said.

I had a similar conversation as a new-ish fractional CTO last year. One team was working on a new CRM product that was effectively alpha-level software used only internally. The team had become terrified of shipping and breaking something and was horrifically risk averse. For a new release that the team was going to delay again at the last minute, I got the CEO on the release call and asked him what would happen if the release completely failed and it took us an entire day to get the product working again. He replied “Not a big deal. The users would just write stuff down like they do today and key it on tomorrow. It’s not like this has enough features to be critical or anything”.

The team was completely stunned. It goes without saying we did the release, found a small mistake, fixed it, and life went on.

Teams really do have understand who her users are and criticality of the software.

Re: Testing on production

#89

Post by chpmrc »

Scubabear68 wrote: post #88
contravariant wrote: post #18

An interesting perspective I once heard from an information security expert is that there's a difference between risks and 'things that can go wrong'. Something is only an actual risk if it hurts the bottom-line. In particular quite a few things that can go wrong don't carry that much risk, and conversely something that is hard but not impossible to go wrong may carry huge amounts of risk. The trick with this perspec…

Very well said. I had a similar conversation as a new-ish fractional CTO last year. One team was working on a new CRM product that was effectively alpha-level software used only internally. The team had become terrified of shipping and breaking something and was horrifically risk averse. For a new release that the team was going to delay again at the last minute, I got the CEO on the release call and asked him what w…

This story perfectly aligns with the arguments in this article. I'll add it as a note if you don't mind.

Re: Testing on production

#90

Post by codethief »

perrygeo wrote: post #60
civilized wrote: post #20

I have a dumb question as a non-SWE who is curious about software engineering. I've heard "feature flags" are popular these days, and I understand that that's where you commit code for a new way of doing things but hide it behind a flag so you don't have to turn it on right away. Now, if I want to test in prod, couldn't I just make the flag for my new feature turn on if I log in on a special developer test account? A…

You've described the ideal use case - a single feature flag, short lived, to let select users test one isolated piece of functionality until it's made generally available. Feature flags used in this way are wonderful. But there are numerous ways to use feature flags incorrectly - typically once you have multiple long-lived flags that interact with each other, you've lost the thread. You no longer have one single appl…

> The only solution is setting up a full integration test for every combination of flags.

I've long been wondering whether there are tools that help with that. Like they measuring a test suite's code coverage but for feature toggle permutations. Either you test those permutations explicitly or you rule them out explicitly.

Post reply on HN