Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

51–60 of 361 posts

Re: My $500M Mars rover mistake

#51
post #6

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

I have bad news for you about health care professionals.

I have worse news for you about grad students.

Re: My $500M Mars rover mistake

#52
post #32

One thing about long aerospace missions like this with huge lead times that always gets me - you can spend years of your life working on a mission, only for it all to fail with potentially years until you can try again. This is a refreshingly humanizing article, but is also one written from the perspective of a survivor. Imagine if the rover were actually lost. I asked the question "what would you do if the mission f…

> "The best thing we can do is learn from failure," one of them told me.

I would argue that if we don't charge the process to prevent this kind of catastrophic failure mode then we really haven't learned from the failure.

Re: My $500M Mars rover mistake

#54
post #42
post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

Of course they are bespoke. There are no COTS Mars rovers for sale.

To be fair; there may be COTS Mars rovers for sale (possibly even cheap), they're just not certified or engineered specifically for Mars. Even an RC car might last long enough for a mission if it was free to get there.

Re: My $500M Mars rover mistake

#55

How is this a $500 million mistake? It seems the issues didn’t cost $500 million.

It's not "mistake" singular, it's actually mistakes.

The first mistake is the $500 million rover is fried.

The second mistake is believing the first mistake.

(or put another way, the first mistake cost $500 million, the second mistake which they didn't realise at the time, saved $500 million)

But you can't explain the second mistake without first explaining the first mistake, hence the title.

Re: My $500M Mars rover mistake

#56
post #30
post #2

So If I understand it correctly, the electrical connector was genderless? If so, that's relatively rare (the only ones I can think of are Anderson Powerpole, which I don't think are rated for interplanetary vehicles) and extremely stupid. Edit: I suppose he could've been using alligator leads.

This was a test fixture so probably banana leads.

I color code _everything_.

Re: My $500M Mars rover mistake

#57

Any test that could cause a fatal destructive error should be risk assesed and a suitable protocol approved with four eyes approval on a final checklist before going hot on the electrics. The issue here is poor project governance not human error.

Yeah, the borrowed multimeter really hammers how silly this all was. You don't touch other peoples lab equipment unless the other ends of the wires are hanging free. A finger pointing to a meter needs to be followed up with a clear confirmation that the wires can be disconnected, and special care isn't needed in the process. If I need something that's connected, I always ask the person to disconnect it for me. Definitely a process/culture problem.

Re: My $500M Mars rover mistake

#58
post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

My understanding is that these electronics need to be radiation hardened (eg; RADHARD) to prevent them from misbehaving in ways far worse than the Belgian bit flipping incident (https://radiolab.org/podcast/bit-flip). If you want to use commercial off the shelf (COTS) parts, you need to install them in triplicate and have them "vote". (https://apps.dtic.mil/sti/citations/ADA391766)

Re: My $500M Mars rover mistake

#60

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…

You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not letting them choose the most effective and low-risk way to do one-off, highly specialized testing work.
Post reply on HN