Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

141–150 of 361 posts

Re: My $500M Mars rover mistake

#141

Post by accrual »

It's a beautiful story. As a space fan, especially of the interplanetary type, this story was riveting to me as it unveiled details of the spacecraft testing I've never imagined. I do far less important testing in my day to day but I was able to draw some similarities with the author.

Many have posted of their failures here so I suppose I could share a couple of mine.

- Pushing gigabytes of records into a Prod table only to realize the primary key was off by a digit, rendering the data useless for a go-live. It had to be deleted by the database admins and reloaded, which took precious hours. I forget why, but an update wasn't feasible.

- A perfect storm of systems issues that lead to all servers in the pool becoming unavailable, causing an entire critical system to go dark. We got it back up within minutes, but harrowing nonetheless.

- Realizing hours before a go-live that a key data element was missing, prompting a client who was now in a code freeze to make a change (they were quite upset). Pretty sure I got an unfavorable review from that, but haven't made the same mistake since.

Re: My $500M Mars rover mistake

#143

Post by frenchie14 »

I worked at JPL for two years in college and helped with flight hardware testing a few times (probably in the same clean room this story took place in, albeit several years later). I can definitely see how a mistake like this could get made. A few stories I remember hearing from those days:

1. Bending pins from trying to insert a connector incorrectly 2. Running a full day of testing but forgetting to set up data recording 3. Accidentally leaving a screwdriver next to the hardware inside a thermal vac chamber in an overnight test

Fun times!

Re: My $500M Mars rover mistake

#144

Post by 6LLvveMx2koXfwn »

markus92 wrote: post #107
lisper wrote: post #85

Earlier quoted context omitted.

Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move i…

How not to check your flight surfaces: Air Astana 1388.

From the Wikipedia article of that flight:

"The incident was featured in season 23, episode 5 of the Canadian documentary series Mayday . . ." [1]

Season 23 - I'm glad I don't fly!

1. https://en.wikipedia.org/wiki/Air_Astana_Flight_1388

Re: My $500M Mars rover mistake

#145

Post by saagarjha »

csomar wrote: post #118
cachvico wrote: post #93

Earlier quoted context omitted.

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

This is bad because if someone forgot to add prod or for whatever reason the code executed beyond the panic, you’ll wipe out the db. There is no code that will protect your db/data. Only replication to a read-only storage will help in such situations.

If code is executing past a panic, I think it is unlikely that you can trust the integrity of your database anyways.

Re: My $500M Mars rover mistake

#146

Post by pants2 »

mk_stjames wrote: post #130

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

And the engineers designing safety mechanisms for nuclear weapons probably think you have it easy.

Re: My $500M Mars rover mistake

#147

Post by amluto »

r2_pilot wrote: post #54
elteto wrote: post #42

Earlier quoted context omitted.

Of course they are bespoke. There are no COTS Mars rovers for sale.

To be fair; there may be COTS Mars rovers for sale (possibly even cheap), they're just not certified or engineered specifically for Mars. Even an RC car might last long enough for a mission if it was free to get there.

Sometimes I wonder whether something like the SpaceX approach would work for these sorts of mission: develop a way to cheaply and reproducibly build the mission hardware, then iterate on it until it works.

Re: My $500M Mars rover mistake

#148

Post by curiousObject »

maxerickson wrote: post #131
curiousObject wrote: post #104

The story is compellingly written, but I thought it was also confusing. It sounds as if this team made several mistakes, not just one mistake. It's also not clear if the result of these mistakes was that there might be real damage to the spacecraft, or if the result was just wasted time and hours of confusion about why the spacecraft wouldn't start up. The first mistake is they didn't realize that the multimeter was…

It started up fine. The multimeter was connecting up the telemetry, so they weren't getting any information from it until they restored that circuit. The power absolutely did feed into that circuit, they were trying to decide if it would have damaged it (but a motor driver is going to be able to handle power coming from the motor, so they decided that it probably didn't damage it).

> It started up fine.

Thanks. I understand that better now. The spacecraft did start up, but it seemed as if it could be badly damaged because they were not receiving any telemetry data

Re: My $500M Mars rover mistake

#149

Post by sopooneo »

maxerickson wrote: post #131
curiousObject wrote: post #104

The story is compellingly written, but I thought it was also confusing. It sounds as if this team made several mistakes, not just one mistake. It's also not clear if the result of these mistakes was that there might be real damage to the spacecraft, or if the result was just wasted time and hours of confusion about why the spacecraft wouldn't start up. The first mistake is they didn't realize that the multimeter was…

It started up fine. The multimeter was connecting up the telemetry, so they weren't getting any information from it until they restored that circuit. The power absolutely did feed into that circuit, they were trying to decide if it would have damaged it (but a motor driver is going to be able to handle power coming from the motor, so they decided that it probably didn't damage it).

That is my reading too. But why was the multimeter connecting up telemetry? That seems very strange to me.

Re: My $500M Mars rover mistake

#150

Post by InSteady »

huytersd wrote: post #103
user_7832 wrote: post #94

Earlier quoted context omitted.

> I guess the idea is that if the problem was incompetency they will find some other way to mess up as opposed to a genuine mistake made by a competent person. Are you suggesting that a competent person never messes up/makes a mistake? The most fundamental part of life is learning from mistakes, and today even AI is starting to do this. Mistakes and evolution are what _make us_ human and living.

Competent people make fewer mistakes because they’re careful and… competent.

Right, but we can all agree that they still make them, and sometimes big ones. Unless they are so careful that they severely limit both their potential and contribution to group. Or they are the rare person who happens to be both "perfect" and lucky.
Post reply on HN