Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

41–50 of 361 posts

Re: My $500M Mars rover mistake

#41

Post by elteto »

krisoft wrote: post #25
elteto wrote: post #9

Earlier quoted context omitted.

So the work is too important to work 12 hour shifts on it, yet your solution is throwing “cheap” grad students at it?

> work is too important to work 12 hour shifts on it Yes. Because it is know that exhausted people make mistakes. The work is too important to let exhausted people screw it up so you should make sure everyone working on it is well rested. > your solution is throwing “cheap” grad students at it Yes? It is testing an electric motor. They can do it. The solution is that you employ enough people so nobody needs to work 1…

That “solution” is nothing more than typical HN backseat driving.

In the real world there are budget, personnel and hiring constraints. You don’t get to hire all the people you want. You make do with what you have, and try to push the mission forward, even in suboptimal conditions.

Re: My $500M Mars rover mistake

#42

Post by elteto »

avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

Of course they are bespoke. There are no COTS Mars rovers for sale.

Re: My $500M Mars rover mistake

#43

Post by pyrolistical »

irjustin wrote: post #16

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…

You should never blame the individual for organizational failures like this. I see two process issues:

1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation

2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it

Blaming individuals only gets them fired and weakens the entire organization. You just fired the one person who learned an expensive lesson.

The only time when an individual should be blamed is when they intended harm, at which case the law could kick in

Re: My $500M Mars rover mistake

#44

Post by [deleted] »

avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

[deleted]

Re: My $500M Mars rover mistake

#45

Post by apcragg »

avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

They almost certainly had flight spares but with two weeks until your launch window, there is zero chance you are deintegrating multiple systems, swapping in the spare, reintegrating, and re running your acceptance test campaigns. And that is assuming that they damaged a subsystem. Back powering the entire spacecraft could have wrecked your power system and anything connected to it. You'd have to disposition every part of the system that was touched. It's much more involved than just swapping in the spare and sending it.

Re: My $500M Mars rover mistake

#46

Post by natosaichek »

avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

I've worked on satellites, and yeah - everything is super bespoke, very low quantity, very expensive. There is probably a qualification unit, or a flight spare that may be available for many subsystems, but maybe not. Integration is a long and complicated process. Pulling apart this bot, with however many fasteners, joints, etc, and then reassembling it correctly would be a decidedly non-trivial project that could easily take a month or two, given that testing has to be performed at each step along the way to ensure that every sensor and actuator is fully functional throughout the integration process. This style of traditional aerospace assembly / integration is not particularly efficient. The only reason it is done this way is that for these kinds of missions you only get one shot, and total cost is ridiculously high so everything must be done correctly.

Re: My $500M Mars rover mistake

#47

Post by CamperBob2 »

spfzero wrote: post #29
kragen wrote: post #14

Earlier quoted context omitted.

maybe it was measuring the current rather than the voltage as he thought, which would require putting it in series with the circuit

That stood out to me as well when I read the article.

The multimeter in the photo has the probes plugged into the volts/ohms jack, though. Maybe it was a different meter.

I figured this was going to be a story about trying to measure voltage with the meter set up on the 10A current range.

Re: My $500M Mars rover mistake

#48

Post by _moof »

avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

Yes, and since they're such low-volume, they are super expensive. I've had to fight tooth and nail just to get more than one set of test hardware.

Re: My $500M Mars rover mistake

#49

Post by ajmurmann »

stickfigure wrote: post #33
lmm wrote: post #6

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

On your next hospital visit for life-saving care, I'm sure you will be comforted to know that nurses (in the US) typically work 12 hour shifts and they're on their feet the whole time.

This is true, but IMO doesn't refute the point, it just makes me concerned about care quality. Were there any studies that showed that the long, grueling shifts are actually better or is it simply this way because it's always been that way and change would be hard and expensive, and because "my grandma walked up hill to school both ways, so young people can too"?

Re: My $500M Mars rover mistake

#50

Post by Aurornis »

stickfigure wrote: post #33
lmm wrote: post #6

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

On your next hospital visit for life-saving care, I'm sure you will be comforted to know that nurses (in the US) typically work 12 hour shifts and they're on their feet the whole time.

Nurses work 3x12 shifts to reduce the number of times patient care is handed off. If everyone worked 8 hour shifts you’d have 3 handoffs per day and a minimum of 3 different people caring for the patient. With 12 hour shifts you have only 2 handoffs per day and can have 2 people trade off on patients if their schedules line up.

I’m sure it varies by location, but my nurse friends only work 3x12, giving them 4 days off per week. Working 12 hour shifts is much more acceptable when you have more days off than days spent working. They’re virtually unavailable on days they work, but then they’re off traveling or having fun for 4 days, some times more if they combine their days off back to back. My close nurse friend routinely takes week long vacations without actually taking any time off at all.

Post reply on HN