Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

61–70 of 361 posts

Re: My $500M Mars rover mistake

#61
post #35

Earlier quoted context omitted.

Wait until I tell you who's doing work in hospitals and how much they're being paid for it.

Another relevant bit of info from hospital accidents: hand-offs between shifts are known to increase the risk of a mistake in care and are part of the reason nurses and doctors work such long hours.

I avoid, if possible of course, going to the hospital right before a shift change for this very reason.

Re: My $500M Mars rover mistake

#62

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…

You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming i…

> only gets them fired

Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.

Re: My $500M Mars rover mistake

#63
post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

I've worked on satellites, and yeah - everything is super bespoke, very low quantity, very expensive. There is probably a qualification unit, or a flight spare that may be available for many subsystems, but maybe not. Integration is a long and complicated process. Pulling apart this bot, with however many fasteners, joints, etc, and then reassembling it correctly would be a decidedly non-trivial project that could ea…

Any idea why they would use brushed motors? When every gram counts I would think ditching the mechanical commutator would be a no-brainer, but maybe adding another leg to the H-bridge is a bigger penalty?

Re: My $500M Mars rover mistake

#64
post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

Your average installed Mars probe widget is not a fungible component; it's been integration tested six ways to Sunday and is certified in its current configuration, as is the assembly of the whole thing, center of gravity, cleanroom status, the torque on every fastener and so on. Even if it's an off the shelf component, it may not be possible to replace it without a chain of sign-offs that would require months of work.

Re: My $500M Mars rover mistake

#65
post #6

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

I have bad news for you about health care professionals.

Health care professionals is a weird one, because while long shifts are dangerous, patient handover is also dangerous and there may be an argument that longer shifts means fewer handovers which could result in better patient outcomes.

Re: My $500M Mars rover mistake

#67
Really well written story.

As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day.

Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had in place was awful and nothing like what we have today—just about every aspect of the process was manual. Anyway, everything was going well. I wrote a few scripts and SQL queries to automate the parts I could. They gave me the production credentials for when I’m ready to deploy. I decided to run what you could call my migration script one last time just to be sure I’m ready. The very moment after I hit the Enter key, I realized I had made a mistake: I had just updated the script with the production credentials just before I made the decision to do another test run. The errors started piling and their service was unresponsive. I was 100% sure I had just wiped their database and I was losing it internally. What saved me was that one of their guys had just a couple hours earlier completed a backup of their database in anticipation of the launch; in the end, they lost a tiny bit of data but most of it was recovered via the backup. Ever since then, “careful” is an extreme understatement when it comes to how I interact with database systems—and production systems in general. Never again.

Re: My $500M Mars rover mistake

#69
post #6

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

One of the hardest problems in a workplace is coordinating the workers. There's also a substantial overhead cost for every employee. I'm sure workers are less efficient at the end of a 12 hour shift, but shift changes also cost a lot and introduce lots of opportunities for errors.

Re: My $500M Mars rover mistake

#70
post #6

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

They definitely should have used a grad student to complete the circuit, freeing up that multimeter.
Post reply on HN