This clickbait title is misleading.
"My almost $500M mistake" doesn't have the same ring.
My $500M Mars rover mistake
191–200 of 361 posts
Re: My $500M Mars rover mistake
#192Earlier quoted context omitted.
You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…
> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.
Re: My $500M Mars rover mistake
#193Earlier quoted context omitted.
You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…
> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.
Re: My $500M Mars rover mistake
#194Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…
Re: My $500M Mars rover mistake
#195Earlier quoted context omitted.
Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update…
That's why I always write a BEGIN statement before executing updates and deletes. If they are not instant or don't return the expected number of modified rows I can just rollback the transaction.
Re: My $500M Mars rover mistake
#196Earlier quoted context omitted.
So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?
I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack. So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some o…
Re: My $500M Mars rover mistake
#197As Fred Rogers said, "look for the helpers".
Re: My $500M Mars rover mistake
#198Earlier quoted context omitted.
You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…
> only gets them fired Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.
Re: My $500M Mars rover mistake
#199Re: My $500M Mars rover mistake
#200Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…
A friend once had to remotely do an OS update of a banking system. Being cautious, he thought he'd back up some central files, just in case and went "mv libc.so old_libc.so". Had to call some guy in that town to throw in the Solaris CD on prem at 2:30 in the morning...