Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

321–330 of 361 posts

Re: My $500M Mars rover mistake

#321

Post by lisper »

kristjansson wrote: post #313
lisper wrote: post #85

Earlier quoted context omitted.

Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move i…

> flight control surfaces move in the right direction How … how often does that go wrong?!

It depends on what you mean by "that". Getting control surfaces actually reversed is not very common, but it does happen, typically after maintenance when a mechanic inadvertently re-connects a control cable backwards.

Control cables also can and do break, but that too is fairly rare.

What is not rare is control mechanisms jamming. Here is an example:

https://www.ntsb.gov/news/press-releases/Pages/NR20230928A.a...

Re: My $500M Mars rover mistake

#322

Post by flyingbarron »

Fantastic story! One of the fascinating things about this are the parallels to Apollo 12 - Lightning strike / Power surge lead to a total loss of telemetry which raised the specter of losing the entire mission - and a similar resolution when telemetry was restored!

I've written a bit about it myself - https://flyingbarron.medium.com/lightning-strikes-92482387ca...

Re: My $500M Mars rover mistake

#323

Post by jdiez17 »

dheera wrote: post #315
jdiez17 wrote: post #301

Earlier quoted context omitted.

Well, it's true that it should be designed such that they cannot be plugged incorrectly. I would imagine it is indeed mostly designed in that way, but there can still be erroneous configurations that were not accounted for at the design stage. Especially during testing you're often dealing with custom cables connectors and circuits that are different from the "normal configuration". I would say that the lesson is to…

> Well, it's true that it should be designed such that they cannot be plugged incorrectly I agree with you, but on Earth this is easy. For spacecraft I imagine you can't just use any connector from Digikey > especially with timeline pressure etc. If timeline pressure, lost sleep, or rushing jobs not meant to be rushed causes a catastrophic technical error to be made, it is 100% the fault of the person who imposed the…

I agree that individuals shouldn't be held responsible for mistakes like this.

However, you can't always eliminate timeline pressure. Even if the project is planned and executed perfectly, there will almost always be unknown unknowns encountered along the way that can push your timeline back. As is the case with sending things to Mars there is a window every two years. That's a very real, non-fictitious deadline that can't be worked around.

Re: My $500M Mars rover mistake

#324

Post by hunterrr »

manfre wrote: post #290
cachvico wrote: post #93

Earlier quoted context omitted.

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

I've added a similar safety to every project. It's not perfect, but this last line of defense has saved team members from themselves more than once. For Django projects, add the below to manage.py: env_name = os.environ.get("ENVIRONMENT", "ENVIRONMENT_NOT_SET") if env_name in TEST_PROTECTED_ENVIRONMENTS and "test" in sys.argv: raise Exception(f"You cannot run tests with ENVIRONMENT={env_name}")

I think runtime checks like this using environment variables is great however what has burned me in the past is that when debugging problems, not knowing what happened at runtime when the logs were produced was problematic. So when test protected environments environment variable needed to be updated I might have a hard time back tracking to it

Re: My $500M Mars rover mistake

#325

Post by Fourier864 »

When I was 20 and in college, I used to work at a Barnes and Noble. When I was working the registers one day, I apparently forgot to put a $300 gift card in a lady's bag at checkout.

The store manager got a complaint about it like a month later, and she tracked down the gift card number from the receipt. It wound up getting loaded with even more money a few days later, and given to someone else.

Anyway, the whole incident got me fired. And to this day, I always check that I put gift cards into bags (on the rare occasion it comes up in my job as a software engineer for NASA missions).

Re: My $500M Mars rover mistake

#326

Post by flashback2199 »

ajmurmann wrote: post #49
stickfigure wrote: post #33

Earlier quoted context omitted.

On your next hospital visit for life-saving care, I'm sure you will be comforted to know that nurses (in the US) typically work 12 hour shifts and they're on their feet the whole time.

This is true, but IMO doesn't refute the point, it just makes me concerned about care quality. Were there any studies that showed that the long, grueling shifts are actually better or is it simply this way because it's always been that way and change would be hard and expensive, and because "my grandma walked up hill to school both ways, so young people can too"?

Who said it's gruelling? In the case of the rover, it's not year round, it's when they are preparing for a launch. Also, you're on HN, so surely you have heard of flow.

Re: My $500M Mars rover mistake

#327

Post by mohas »

So let me share mine, it goes like hey can you correct this issue directly on the database of 5 years of corporate financial data? and in midst of testing and looking for the issue I wrote a delete with incomplete where clause that would delete lots of data, fortunately it was stopped because of a constraint violation, I still remember the adrenaline rush when I was processing what just happened

Re: My $500M Mars rover mistake

#328

Post by xp84 »

mango7283 wrote: post #243
hutzlibu wrote: post #240

Earlier quoted context omitted.

The suspicious part for me would be the CEO laughing like it was nothing. Also yes, one would expect it goes the other way around, when you messed up big, someone will come to you. But the world is big and maybe it happened like this.

Surely this is a variation of this anecdote attributed to IBM's Watson https://news.ycombinator.com/item?id=13419313 "> A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to ha…

In the context of “suspected AI” I at first thought you meant a different Watson!

Re: My $500M Mars rover mistake

#329

Post by cpgxiii »

wolverine876 wrote: post #187
cpgxiii wrote: post #132

Earlier quoted context omitted.

In 2003, small brushless DC motors were way less mature and available as they are now, particularly for low-speed/high-torque applications*. Brushless controllers are much more complicated than brushed controllers, particularly on the control software front, so sticking with simpler and more reliable brushed controllers for a space application makes sense (remember, it probably needed to be radiation hardened - doing…

Without disagreeing with your point, would availability be an issue in this case? They need one or two, have an enormous budget, and if the technology exists can make their own.

Availability is often strongly correlated with technical maturity. Small brushless motors with FOC didn't become widely available and mature until really the late 2010s. Arguably the foundation of nearly all of DJI's product lines is due to their early mastery of small brushless motor control (drones, gimbals, lens controls, robots, etc), and that's a company founded in 2006, well after the events of the article.

You can get good control of brushed motors with just a couple of transistors. Good brushless control means FOC, which really requires a fairly capable microcontroller in addition to all the power electronics for variable-frequency drive. While brushed motors certainly have limitations, those were quite well understood by the early 2000s (to the point here that assessing whether or not damage had occurred was "just" a question of "have these few transistors suffered from voltage applied in an unintended manner"). Brushless motors involve way more components with way more integration required to make then small. Far more complexity and potential failure modes need to be understood.

Re: My $500M Mars rover mistake

#330

Post by dheera »

jdiez17 wrote: post #323
dheera wrote: post #315

Earlier quoted context omitted.

> Well, it's true that it should be designed such that they cannot be plugged incorrectly I agree with you, but on Earth this is easy. For spacecraft I imagine you can't just use any connector from Digikey > especially with timeline pressure etc. If timeline pressure, lost sleep, or rushing jobs not meant to be rushed causes a catastrophic technical error to be made, it is 100% the fault of the person who imposed the…

I agree that individuals shouldn't be held responsible for mistakes like this. However, you can't always eliminate timeline pressure. Even if the project is planned and executed perfectly, there will almost always be unknown unknowns encountered along the way that can push your timeline back. As is the case with sending things to Mars there is a window every two years. That's a very real, non-fictitious deadline that…

> As is the case with sending things to Mars there is a window every two years.

This is very simple to deal with.

(a) If it's unmanned, rush and launch on-time but don't fault the engineer for mistakes made by rushing. If it doesn't work everyone accept that as a consequence of rushing.

(b) If it's manned, wait until the next launch window and prioritize safety. Period.

Post reply on HN