Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

191–200 of 361 posts

Re: My $500M Mars rover mistake

#191

This clickbait title is misleading.

"My almost $500M mistake" doesn't have the same ring.

"My mistake that seemed to potentially cost up to $500M, but then it turned out fine" would be more accurate. While a more appropriate title would be something akin to "learning from mistakes" or "a Mars rover testing scare".

Re: My $500M Mars rover mistake

#192
post #62

Earlier quoted context omitted.

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…

> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.

Firing somebody for a simple mistake with grave consequences doesn't make your organization stronger. There will be plenty of better examples to make.

Re: My $500M Mars rover mistake

#193
post #62

Earlier quoted context omitted.

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…

> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.

Firing people for incidental mistakes instead of overall bad performance is pretty shitty management.

Re: My $500M Mars rover mistake

#194

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

A friend once had to remotely do an OS update of a banking system. Being cautious, he thought he'd back up some central files, just in case and went "mv libc.so old_libc.so". Had to call some guy in that town to throw in the Solaris CD on prem at 2:30 in the morning...

Re: My $500M Mars rover mistake

#195
post #182
post #157

Earlier quoted context omitted.

Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update…

That's why I always write a BEGIN statement before executing updates and deletes. If they are not instant or don't return the expected number of modified rows I can just rollback the transaction.

[deleted]

Re: My $500M Mars rover mistake

#196
post #115

Earlier quoted context omitted.

So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?

I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack. So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some o…

One guy in our workshop had to provide DC to a display with a round 4-pin connector. He soldered two neighboring pins to Gnd and the other two to Vcc. There were two chances to short the powersupply, one to brick the display and one to get it right. Guess what we had to replace until we found out.

Re: My $500M Mars rover mistake

#198
post #62

Earlier quoted context omitted.

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…

> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.

So you'd not blame them for their simple mistake, but still fire them for it?

Re: My $500M Mars rover mistake

#199
This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the crucial variables was blank in production. I could have expected this if I had spent more time preparing and reading documentation. Sales died completely, and my heart sank. After a lengthy rollback procedure that resulted in a few hours without sales, a massive surge of angry customers, and a loss of several million euros, I approached the CEO of the company. I still remember catching him in an elevator. I explained that this incident was all my fault and I had failed to properly analyse the environment. I assured him that I was ready to bear full consequences, including being fired. He burst into laughter and said something like this: "Why would I want to get rid of you? You made a mistake that you'll never do again. You are better at your job than you were yesterday!" This experience was formative to me on many levels including true leadership. I successfully completed many high risk projects since than.

Re: My $500M Mars rover mistake

#200

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

A friend once had to remotely do an OS update of a banking system. Being cautious, he thought he'd back up some central files, just in case and went "mv libc.so old_libc.so". Had to call some guy in that town to throw in the Solaris CD on prem at 2:30 in the morning...

Never this simple and calling someone still probs right thing to but fixing stuff like this is what /sbin is for.
Post reply on HN