Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

181–190 of 361 posts

Re: My $500M Mars rover mistake

#181

Earlier quoted context omitted.

They fired you for that array_reverse mistake?

Yep, I was put on PIP with impossible success criteria (no issues raised in PRs by senior engineers and no issues in code deployed to production - even if it was reviewed by senior engineers & QA) and fired (for failing that criteria) in 2 weeks. I worked there for ~8 months in total.

The impossible PIP trick for dismissal is something I’d love to see get eventually legally obliterated.

Re: My $500M Mars rover mistake

#182
post #157

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update…

That's why I always write a BEGIN statement before executing updates and deletes. If they are not instant or don't return the expected number of modified rows I can just rollback the transaction.

Re: My $500M Mars rover mistake

#183
I used to work in a rural manufacturing plant. I once totaled my car on the way to town. Our shop foreman was also a volunteer firefighter and was the first one on the scene. I survived with a bloody nose and shattered confidence. The next day at work we needed some parts from town. I didn't have a car anymore, but that same shop foreman lent me his suped up pickup truck. I was totally confused. He wisely said:

"Today is the safest day to let you drive my truck, cause I know you'll be extra careful"

Re: My $500M Mars rover mistake

#184

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…

At a major brokerage firm I accidentally hit prod with a testing script that did several millions of dollars of fake FX test trades. The first thing mentioned in the post mortem call was “No one is going to blame the guy who did those trades. It was an honest mistake. What we are going by to do is discuss why a developer can hit the production trading API without any authentication at all”.

Were the trades any good though?

Re: My $500M Mars rover mistake

#185

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

Regarding your last paragraph, I thought the same... When the author wrote:

> I'm instantly transported back to that moment — the room, the lighting, the chair I was in, the table, the pit in my stomach, ...

I could'nt help but think "that sounds like a trauma reaction". Good on them to be able to use that energy to do better! But also not everyone reacts the same way to trauma nor is it easy to compare such reactions to trauma (for example as a hiring question). I feel there are too many social variables at play

Re: My $500M Mars rover mistake

#186

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

> I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. Does it inevitably come down to that for someone? I mean even if its a detail that a procedure couldn’t have caught, someone is responsible for forming good procedures. I suppose there could be several factors. But it seems like ultimately someone is going to be pretty directly resp…

"But it seems like ultimately someone is going to be pretty directly responsible."

Or many people, or no one directly. Space missions come with calculated risk. So someone calculates the risk that this critical part brakes is 0.5% and then someone higher up says, that is acceptable and all move on - and then this part indeed brakes and people die.

Who is to blame, when the calculation was indeed correct, but 0.5% chances still can happen (and itnwould be a lot)? And economic pressures are real, like the limits of physic?

See Murpheys Law, "Anything that can go wrong will go wrong." (Eventually, if done again and again)

https://en.m.wikipedia.org/wiki/Murphy's_law

Astronauts know, there is a risk with every mission, so do the engeneers, so does management. Still, I cannot imagine why anyone thought it was an acceptable risk, to use a 100% oxygen atmosphere with Apollo 1, where 3 Astronauts died in a fire. But that incident indeed changed a lot regarding safety procedures and thinking about safety. Still, some risk remain and you have to live with that.

I am quite happy though, that in my line of work, the worst that can happen is a browser crash.

Re: My $500M Mars rover mistake

#187
post #63

Earlier quoted context omitted.

Any idea why they would use brushed motors? When every gram counts I would think ditching the mechanical commutator would be a no-brainer, but maybe adding another leg to the H-bridge is a bigger penalty?

In 2003, small brushless DC motors were way less mature and available as they are now, particularly for low-speed/high-torque applications*. Brushless controllers are much more complicated than brushed controllers, particularly on the control software front, so sticking with simpler and more reliable brushed controllers for a space application makes sense (remember, it probably needed to be radiation hardened - doing…

Without disagreeing with your point, would availability be an issue in this case? They need one or two, have an enormous budget, and if the technology exists can make their own.

Re: My $500M Mars rover mistake

#188
post #183

I used to work in a rural manufacturing plant. I once totaled my car on the way to town. Our shop foreman was also a volunteer firefighter and was the first one on the scene. I survived with a bloody nose and shattered confidence. The next day at work we needed some parts from town. I didn't have a car anymore, but that same shop foreman lent me his suped up pickup truck. I was totally confused. He wisely said: "Toda…

You'll be extra careful but I don't know about extra safe. I had a few close calls while driving and it made me a less confident driver which I don't think did anyone any favors.

Re: My $500M Mars rover mistake

#189

Bravo to the author. I think if I had made the same mistake, and if the disaster is permanent, I would probably be scarred for life and never recover from it.

The author is scarred for life, reliving it in certain circumstances. They use the scar as a tool to sharpen their performance.

Re: My $500M Mars rover mistake

#190

Earlier quoted context omitted.

They fired you for that array_reverse mistake?

Yep, I was put on PIP with impossible success criteria (no issues raised in PRs by senior engineers and no issues in code deployed to production - even if it was reviewed by senior engineers & QA) and fired (for failing that criteria) in 2 weeks. I worked there for ~8 months in total.

> no issues raised in PRs by senior engineers

Wat? Like serious issues, or minor things that can be improved? Because it's very rare in my place of work that there are no comments on a 'PR'. Something can always be improved.

Post reply on HN