My $500M Mars rover mistake • Hacker News Board

Re: My $500M Mars rover mistake

#301

Post by jdiez17 » Wed, Nov 29, 2023, 3:30 PM UTC

i-use-nixos-btw wrote: post #294

I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them cus…

Well, it's true that it should be designed such that they cannot be plugged incorrectly. I would imagine it is indeed mostly designed in that way, but there can still be erroneous configurations that were not accounted for at the design stage.

Especially during testing you're often dealing with custom cables connectors and circuits that are different from the "normal configuration".

I would say that the lesson is to do as many critical operations under the 4-eye principle: someone is doing the thing, someone else is checking each step before continuing. Very effective for catching "stupid mistakes" like the one in the article. But again, it is not always possible to have two people looking at one test, especially with timeline pressure etc. So mistakes like these do happen in the real world. You have to make the whole system robust.

Re: My $500M Mars rover mistake

#302

Post by Retric » Wed, Nov 29, 2023, 3:37 PM UTC

crazygringo wrote: post #299

i-use-nixos-btw wrote: post #294

I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them cus…

That was my first thought as well. On the other hand, it's hard to make these kinds of judgment calls when you're talking about a one-off piece of equipment that's only going to go through this particular testing cycle a single time. In computing, there are a lot of similar "one-off" operations -- something you to do to the prod database or router config a single time as part of an upgrade or migration. Sometimes bui…

“One off” is never just a one off it’s always part of a class of activity such as server migrations etc. Just paying attention guarantees eventual failure when repeated enough times.

This may be acceptable, but it comes down to managing risks. If failure means the company dies then taking a 1 in 10,000 risk to save 3 hours of work probably isn’t worth it. If failure means an extra 100 ours of work and 10k in lost revenue then sure take that 1 in 10,000 risk it’s a reasonable trade off.

Re: My $500M Mars rover mistake

#303

Post by Tangurena2 » Wed, Nov 29, 2023, 3:48 PM UTC

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

One way I mistake-proof things in SQL Management Studio is to have different colors for production vs test databases.

To do that, on the "connect to server" dialog, click "options". On the tab "connection properties" in the "connection" option group, check "use custom color". And I pick the reddest red there is. The bottom of the results window will have that color.

edit: my horrible foul-up was restoring a database to production. The "there is trouble" pagers were all Iridium pagers since they loved climbing mountains (where there was no cell service back then). But then that place didn't use source control, so it was disasters all the way down.

Re: My $500M Mars rover mistake

#304

Post by dimal » Wed, Nov 29, 2023, 3:48 PM UTC

> failure is not an option — it comes pre-installed

Love this. Whenever people say “failure is not an option”, I get the sense that they don’t really understand how the universe works. It’s like saying “entropy is not an option”. Uh...

Re: My $500M Mars rover mistake

#305

Post by lbriner » Wed, Nov 29, 2023, 4:02 PM UTC

There is a principle here that I can't see mentioned and that is how easy it is to discard something as a cause of a problem because it seems so minor, so routine (i.e. removing the "spare" multimeter) and therefore getting blinkered on what is going on.

There are more than a few times where I am scratching my head as to "how could my change have possibly broken this" only to remember a couple of hours later that I had made another change somewhere, or rebooted, or changed a config file temporarily.

I guess it just says that we all need to log everything we do, including removing spare multi-meters, so that by looking over the list we can remember these things.

Re: My $500M Mars rover mistake

#306

Post by zimpenfish » Wed, Nov 29, 2023, 4:09 PM UTC

hannofcart wrote: post #264

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

What strikes me as remarkable in all such stories is how almost always, the person committing the mistake is a junior who never deserves the blame. And how cavalier the handoff/onboarding by the 'seniors' working on the projects are. Having worked in enough of these though, I am aware that even they (the "seniors") are seldom entirely responsible for all the issues. It's mostly business constraints that forces cuttin…

As I said on Slack the other day in response to a similar story, "If, on your first day, you can destroy the prod database, it's not your fault."

(One of my standard end-of-interview questions is "how easy is it for me to trash the production database?" Having done this previously[1] and had a few near misses, it's not something I want to do again.)

[1] In my defence, I was young and didn't know that /tmp on Solaris was special. Not until someone rebooted the box, anyway.

Re: My $500M Mars rover mistake

#307

Post by Dioxide2119 » Wed, Nov 29, 2023, 4:15 PM UTC

astrolx wrote: post #241

idlewords wrote: post #60

Earlier quoted context omitted.

You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not lett…

I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present,…

The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it.

"It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone.

I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.

Re: My $500M Mars rover mistake

#308

Post by Saris » Wed, Nov 29, 2023, 4:15 PM UTC

frankus wrote: post #63

natosaichek wrote: post #46

Earlier quoted context omitted.

I've worked on satellites, and yeah - everything is super bespoke, very low quantity, very expensive. There is probably a qualification unit, or a flight spare that may be available for many subsystems, but maybe not. Integration is a long and complicated process. Pulling apart this bot, with however many fasteners, joints, etc, and then reassembling it correctly would be a decidedly non-trivial project that could ea…

Any idea why they would use brushed motors? When every gram counts I would think ditching the mechanical commutator would be a no-brainer, but maybe adding another leg to the H-bridge is a bigger penalty?

Closed loop control of brushless motors is just more complex, in addition to needing 3 phase AC output, you also need either hall sensors or an encoder of some kind to be able to start the motor smoothly, and you need a dedicated IC or MCU for each motor to manage commutation and read the sensors.

I don't think FOC type controllers were anywhere near common back then either, which is needed to run a brushless motor smoothly.

There is just so much more that can go wrong with a brushless setup, vs brushed where you just apply power and that's it.

Re: My $500M Mars rover mistake

#309

Post by arethuza » Wed, Nov 29, 2023, 4:17 PM UTC

markmark wrote: post #160

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each…

About 10 years ago I literally saw the blood drain from a colleagues face as he realised he had dropped a production database because he thought he was in a dev environment.

A DBA colleague sitting nearby laughed and had things restored back within a few minutes....

Re: My $500M Mars rover mistake

#310

Post by alibarber » Wed, Nov 29, 2023, 4:20 PM UTC

i-use-nixos-btw wrote: post #294

I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them cus…

| The "trust the 28 year old with the $500m robot that can go boom if they slip up" logic seems very peculiar.

Not just that, but to create a situation whereby said person is working unofficial double shifts to get it done, so probably aren't going to be bringing their best selves into the office. If it were my $500 million I wouldn't even care about the name of this guy but would want to have some very robust discussions with the head of their department. Also, "some mistakes feel worse than death" - I get it, but c'mon, it's not like someone actually did die, which is a sadly unfortunate reality of other much less spectacular and blog-worthy mistakes.