My $500M Mars rover mistake • Hacker News Board

Re: My $500M Mars rover mistake

#271

Post by ThePowerOfFuet » Wed, Nov 29, 2023, 12:16 PM UTC

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

>The very moment after I hit the Enter key, I realized I had made a mistake

This brief moment in time has a name: an ohnosecond.

https://en.wiktionary.org/wiki/ohnosecond

Re: My $500M Mars rover mistake

#272

Post by ThePowerOfFuet » Wed, Nov 29, 2023, 12:17 PM UTC

Feathercrown wrote: post #169

markmark wrote: post #160

Earlier quoted context omitted.

At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each…

Isn't that almost exactly what happened at github too?

Yup.

Re: My $500M Mars rover mistake

#273

Post by HeavyStorm » Wed, Nov 29, 2023, 1:04 PM UTC

It's a interesting story, but the author may be overselling it. It's not a _failure_ story, nor was it a 500M mistake. I get that it was really stressful and the mistake could have cost him the job, but it didn't; it also didn't cost NASA anything other than a few hours of work (which, during testing, I would guess it's expected).

When I'm asked to share failures, I'm usually not thinking about "that one time when I almost screwed up but everything was fine", instead, I'm thinking of when I actually did damage to the business and had to fix it somehow.

Re: My $500M Mars rover mistake

#274

Post by hutzlibu » Wed, Nov 29, 2023, 1:04 PM UTC

me_me_me wrote: post #245

hutzlibu wrote: post #240

Earlier quoted context omitted.

The suspicious part for me would be the CEO laughing like it was nothing. Also yes, one would expect it goes the other way around, when you messed up big, someone will come to you. But the world is big and maybe it happened like this.

Story is real or not aside, why would you not laugh it off? At that stage nothing can be changed, money was lost and bug was fixed. You can only look forward and plan for the future and the guy is going to be paranoid in the future deployments to make sure not to fuck up again.

In general yes, and people with enough Zen can do this. But if also the CEO is looking forward to hear from the board and the investors to explain the incident, he might not be in the mood to laugh.

Re: My $500M Mars rover mistake

#275

Post by elteto » Wed, Nov 29, 2023, 1:05 PM UTC

noSyncCloud wrote: post #73

elteto wrote: post #61

Earlier quoted context omitted.

I avoid, if possible of course, going to the hospital right before a shift change for this very reason.

I'm not questioning your logic here, but how do you keep intimate knowledge of the seasonal vagaries of shift changes at every department of your local hospital?

If you’ve been to that hospital you can easily take note. And most places have a shift change between 5-7am. This one is almost universal, as far as I have observed, even in different countries.

Re: My $500M Mars rover mistake

#276

Post by numpad0 » Wed, Nov 29, 2023, 1:09 PM UTC

hutzlibu wrote: post #270

creshal wrote: post #254

Earlier quoted context omitted.

A variant is in From the Earth to the Moon , where a junior engineer at Grumman confesses messing up vital calculations for the Lunar Lander to his boss, and finishes with "So… I guess I'll go clean out my desk." "What for?" "I figure you're gonna fire me now." The boss's response makes a lot more sense than the usual fluff, though: "If I fire you now, the next guy to make a mistake won't admit it and we won't find o…

I wonder how much of those stories are rather wishful thinking of how it should work and not how it does work, when a major screw up happened and some heads need to roll for the sake of it.

I wonder how would the boss explain it to his bosses/shareholders. Was that totally a known possible outcome that merely surfaced by chance and subsequently handled without issues under his leadership, or...?

Re: My $500M Mars rover mistake

#277

Post by atemerev » Wed, Nov 29, 2023, 1:13 PM UTC

Ctrl-F, "The Martian"

"Phrase not found"

But... how? This is exactly the same story that happened in the book. Since Spirit is older is the book, perhaps this was the real-life inspiration.

Re: My $500M Mars rover mistake

#278

Post by creshal » Wed, Nov 29, 2023, 1:33 PM UTC

numpad0 wrote: post #276

hutzlibu wrote: post #270

Earlier quoted context omitted.

I wonder how much of those stories are rather wishful thinking of how it should work and not how it does work, when a major screw up happened and some heads need to roll for the sake of it.

I wonder how would the boss explain it to his bosses/shareholders. Was that totally a known possible outcome that merely surfaced by chance and subsequently handled without issues under his leadership, or...?

Apollo was very much pushing the envelope of bleeding edge technology, while the bosses were probably not too happy, it was far from the only occurrence, and didn't threaten the contract.

Re: My $500M Mars rover mistake

#279

Post by NL807 » Wed, Nov 29, 2023, 1:33 PM UTC

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

Hats off to the backup guys.

Re: My $500M Mars rover mistake

#280

Post by mosburger » Wed, Nov 29, 2023, 1:46 PM UTC

Hey neat - I once destroyed a $5000 prototype disk drive (I was a Firmware engineer at Quantum in the late 1990s at my first post-college job), also with back-EMF (it flowed from the braking spindle motor into a power chip and melted the circuit board). I felt terrible and that was multiple orders of magnitude cheaper. :)