Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

341–350 of 361 posts

Re: My $500M Mars rover mistake

#341

Post by lucb1e »

bboygravity wrote: post #203
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

You worried about that? I'm a frequent flyer and I got a feeling that most airline ticket booking pages are broken in some way more than half the time. Maybe not often broken to the point that they're blank, but definitely broken to the point that booking a ticket isn't possible (I prefer blank, so that I don't waste like 30 minutes on not being able to book a ticket). Also most of the internet seems often broken. Oh…

Woa, and I always wonder why it's only me that seems to have to use the developer tools to enable that stupid submit button when I filled out every field on the page correctly, shaking my head and wondering how normal people use the internet. I keep thinking it's got to be something about using firefox instead of a big tech browser, my mouse gestures extension, I don't know but normal webshops are broken so often it's insane. Thanks for sharing that it's not just me!

Re: My $500M Mars rover mistake

#342

Post by lucb1e »

i-use-nixos-btw wrote: post #294

I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them cus…

> The "trust the 28 year old

(Same for an 82 year old or any other number..)

Re: My $500M Mars rover mistake

#343

Post by sbierwagen »

gosub100 wrote: post #331
lostlogin wrote: post #318

Earlier quoted context omitted.

> /tmp on Solaris was special. I’ve had a search but can’t work out why it’s special.

it gets wiped on reboot. I remember around 2007 on Gentoo Linux, this behavior changed. I was using /tmp as pretty much a "my documents" type folder, I updated, and one day all my stuff was gone! I was flabbergasted. But yeah, it was reckless to store things on a folder that pretty has "temp" in the name!

This is rude, but I'd like to reply a comment you deleted in a separate thread.

"why didn't they have a hot-spare" They do! Flight spares are complete, flight-rated copies of spacecraft built for exactly this contingency: https://en.wikipedia.org/wiki/Flight_spare After launch the flight spares are used for terrain testing and troubleshooting. (The "mars yard" has flight spares for Curiosity and Perseverance https://www-robotics.jpl.nasa.gov/how-we-do-it/facilities/ma... which were used to test some wheels to destruction after Curiosity started showing some wear https://www.planetary.org/articles/08190630-curiosity-wheel-... )

The blog post lays it on a bit thick with the $500 million number and the "launch only two weeks away" given that the article itself is illustrated with a photo of the Sojourner flight spare. Spirit had the SSTB1 test rover. If he had actually blown out the entire electrical system, they could have launched it instead. Swapping out the entire vehicle right before launch would have been an awful job, but it's not flat out impossible.

Re: My $500M Mars rover mistake

#344

Post by sbierwagen »

apcragg wrote: post #45
avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

They almost certainly had flight spares but with two weeks until your launch window, there is zero chance you are deintegrating multiple systems, swapping in the spare, reintegrating, and re running your acceptance test campaigns. And that is assuming that they damaged a subsystem. Back powering the entire spacecraft could have wrecked your power system and anything connected to it. You'd have to disposition every pa…

Spirit/Opportunity had the SSTB1 test rover, which supposedly had a complete set of scientific instruments. If it was fully qualified and tested, swapping it out could have been as easy as dropping it in the lander and writing a different serial number in the paperwork.

(I really doubt it was fully tested. But why else have a flight spare vehicle?)

Re: My $500M Mars rover mistake

#345

Post by ahazred8ta »

kristjansson wrote: post #313
lisper wrote: post #85

Earlier quoted context omitted.

Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move i…

> flight control surfaces move in the right direction How … how often does that go wrong?!

There have been several cases of the landing gear up/down lever getting wired backwards during maintenance. Not to worry, the gear has a 'squat switch' sensor that prevents the gear from being raised when the plane is on the ground. Unless you taxi over a bump and the switch decides it's now airborne. Crunch.

Re: My $500M Mars rover mistake

#346

Post by p_plno »

I'm really at a loss of words. There is only one lesson from this incident: nobody is supposed to touch a 500M piece of equipment after 12 hours of work. Period. The author is not getting it, and nobody in the comments. The world is run by idiots and it shows.

Re: My $500M Mars rover mistake

#347

Post by RHSman2 »

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

I’d love to know the long term physiological effect on the body of these events. Have had a few. Still feel shakey :)

Re: My $500M Mars rover mistake

#348

Post by astrolx »

Dioxide2119 wrote: post #307
astrolx wrote: post #241

Earlier quoted context omitted.

I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present,…

The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it. "It's a waste of time" is very often a fallacy, especially when…

Exactly :). The funny part is, the thing actually crashed! [1]

Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie !

[1] https://exploration.esa.int/web/mars/-/59176-exomars-2016-sc...

Re: My $500M Mars rover mistake

#349

Post by zilti »

astrolx wrote: post #241
idlewords wrote: post #60

Earlier quoted context omitted.

You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not lett…

I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly. The simplest task you can imagine takes incredible proportions (for good reasons). Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present,…

And we all know how reliable ESA landers are. The laughing stock of the industry.

Re: My $500M Mars rover mistake

#350

Post by gosub100 »

sbierwagen wrote: post #343
gosub100 wrote: post #331

Earlier quoted context omitted.

it gets wiped on reboot. I remember around 2007 on Gentoo Linux, this behavior changed. I was using /tmp as pretty much a "my documents" type folder, I updated, and one day all my stuff was gone! I was flabbergasted. But yeah, it was reckless to store things on a folder that pretty has "temp" in the name!

This is rude, but I'd like to reply a comment you deleted in a separate thread. "why didn't they have a hot-spare" They do! Flight spares are complete, flight-rated copies of spacecraft built for exactly this contingency: https://en.wikipedia.org/wiki/Flight_spare After launch the flight spares are used for terrain testing and troubleshooting. (The "mars yard" has flight spares for Curiosity and Perseverance https://…

Not rude at all! I appreciate the reply. Only reason I deleted my message was because right after posting, I scrolled down and saw someone asking the exact same question at the top level, so I felt like it was best to conserve effort and not repeat them.

I liked that other people pointed out that risk could have been eliminated by using polarized connectors (I hope they started doing this after the incident), but also made me wonder about "back-EMF" caused by solar flares. In other words, maybe all thick wires and ground/power planes should be hardened against current surges simply due to a solar event hitting mars (which may incidentally cover the case of back-powering the driver circuits).

Post reply on HN