Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

351–360 of 361 posts

Re: My $500M Mars rover mistake

#351

Post by hn_go_brrrrr »

Scubabear68 wrote: post #89
irjustin wrote: post #16

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…

At a major brokerage firm I accidentally hit prod with a testing script that did several millions of dollars of fake FX test trades. The first thing mentioned in the post mortem call was “No one is going to blame the guy who did those trades. It was an honest mistake. What we are going by to do is discuss why a developer can hit the production trading API without any authentication at all”.

That's an excellent postmortem culture.

Re: My $500M Mars rover mistake

#352

Post by HenriTEL »

askvictor wrote: post #109
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

One place I worked (some 20 years ago) had a policy that any time you run a sudo command, another person has to check the command before you hit enter. Could apply the same kind of policy/convention for anything in production.

That's not a good advice IMO, as most sudo commands will mess-up just one host, and it's something you should generally be prepared for. You're more likely to develop a culture where engineers think about hosts as critical resources whereas they should be generally considered as instances that can be thrown away. It's better to identify hosts that are SPOF and be cautious on those only.

I can think of a larger blast radius when deleting files on a shared mount point for example but it's not representative to the regular use of sudo.

Re: My $500M Mars rover mistake

#354

Post by HenriTEL »

fransje26 wrote: post #248

> I was into my unofficial second shift having already logged 12 hours that Wednesday. Long workdays are a nominal scenario for the assembly and test phase. Although the time pressure coming with the upcoming deadline is understandable, perhaps the bigger lessons here is that when you are possibly sleep-deprived, and have already pulled too long a shift, you are bound to make avoidable mistakes. And that is the last…

Yep. Beyond the technical issue this story shows a people management issue.

Re: My $500M Mars rover mistake

#355

Post by HenriTEL »

p_plno wrote: post #346

I'm really at a loss of words. There is only one lesson from this incident: nobody is supposed to touch a 500M piece of equipment after 12 hours of work. Period. The author is not getting it, and nobody in the comments. The world is run by idiots and it shows.

It seems like there was a lot of pressure to meet the deadline and everybody overworked in this environment. A perfect example of normalization of deviance. It's a shame that there still exists many workplaces like that. It's our duty to be aware of what's happening around us and tell the guy - Hey, what the fuck are you doing here? Get home, that's no critical emergency.

Re: My $500M Mars rover mistake

#356

Post by lostlogin »

gosub100 wrote: post #331
lostlogin wrote: post #318

Earlier quoted context omitted.

> /tmp on Solaris was special. I’ve had a search but can’t work out why it’s special.

it gets wiped on reboot. I remember around 2007 on Gentoo Linux, this behavior changed. I was using /tmp as pretty much a "my documents" type folder, I updated, and one day all my stuff was gone! I was flabbergasted. But yeah, it was reckless to store things on a folder that pretty has "temp" in the name!

Thanks you.

I have been burned by this in some version of Ubuntu and have assumed it was normal behaviour ever since.

Re: My $500M Mars rover mistake

#357

Post by cloths »

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

We used to have another engineer watch over your shoulder when you do Prod stuff, can be very helpful.

Re: My $500M Mars rover mistake

#358

Post by hnarayanan »

KMag wrote: post #269
hnarayanan wrote: post #239

Earlier quoted context omitted.

Plot twist: It made so much money that that’s now their strategy.

Back in school, my roommate's mom worked for a hedge fund and he did part-time work for them. He factored out a common trading engine from individual strategies, and one day the head of the fund asked him to run a strategy that had made a bunch of money in the past, but had been retired after failing to make money for a while. So, he put the strategy back in production without any testing, forgetting that he had rece…

This is such a cool story.

Re: My $500M Mars rover mistake

#360

Post by chrz »

supriyo-biswas wrote: post #234
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

The language is just so anodyne and there’s just that bit of implausible detail in the story (approaching the CEO yourself when you’re the one who fucked up, also how parent claims to be a “top performer” and “I made my company lose millions” at the same time) makes me think this comment was written by an LLM, or at least a fabrication.

Also airlines doesnt sell millions of $ in tickets every day
Post reply on HN