Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

281–290 of 361 posts

Re: My $500M Mars rover mistake

#281

Post by numpad0 »

maxerickson wrote: post #95
lisper wrote: post #85

Earlier quoted context omitted.

Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move i…

So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?

Looks like the author tried to double-power the motor with both the spacecraft motor driver and through the breakout box that MITMs the driver and the motor. In such event, the free-wheeling diode in the driver will allow reverse current to be fed back to the driver's power supply up to certain amounts. This will absorb back-EMF, or energy from "regen" from the motor.

I'm suspecting the breakout wasn't literally sitting between the driver and the motor, but rather all internal connections are broken out to the box for testing; and likely the author's mistake was to not mess with the spacecraft to temporarily disconnect the driver.

But I'm not sure if I'd "just" made the right call and done so nonchalantly on a Mars rover to launch in few weeks.

Re: My $500M Mars rover mistake

#282

Post by felideon »

HeavyStorm wrote: post #273

It's a interesting story, but the author may be overselling it. It's not a _failure_ story, nor was it a 500M mistake. I get that it was really stressful and the mistake could have cost him the job, but it didn't; it also didn't cost NASA anything other than a few hours of work (which, during testing, I would guess it's expected). When I'm asked to share failures, I'm usually not thinking about "that one time when I…

“My quasi-$500M Mars rover mistake” just doesn’t have quite the same ring to it.

But, to your point, it is still a failure story that _could_ have lead to a much worse outcome than it did. The fact that it didn’t was mostly due to luck.

Re: My $500M Mars rover mistake

#283

Post by euroderf »

hutzlibu wrote: post #186
nonethewiser wrote: post #158

Earlier quoted context omitted.

> I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. Does it inevitably come down to that for someone? I mean even if its a detail that a procedure couldn’t have caught, someone is responsible for forming good procedures. I suppose there could be several factors. But it seems like ultimately someone is going to be pretty directly resp…

"But it seems like ultimately someone is going to be pretty directly responsible." Or many people, or no one directly. Space missions come with calculated risk. So someone calculates the risk that this critical part brakes is 0.5% and then someone higher up says, that is acceptable and all move on - and then this part indeed brakes and people die. Who is to blame, when the calculation was indeed correct, but 0.5% cha…

> Still, I cannot imagine why anyone thought it was an acceptable risk, to use a 100% oxygen atmosphere with Apollo 1

Especially when, in prior experience there, asbestos had caught fire in the same situation (O2, low pressure).

Re: My $500M Mars rover mistake

#284

Post by pbj1968 »

noSyncCloud wrote: post #73
elteto wrote: post #61

Earlier quoted context omitted.

I avoid, if possible of course, going to the hospital right before a shift change for this very reason.

I'm not questioning your logic here, but how do you keep intimate knowledge of the seasonal vagaries of shift changes at every department of your local hospital?

Because they’re more or less universal.

Re: My $500M Mars rover mistake

#285

Post by JoeAltmaier »

My friend reports his phone started dropping contacts methodically, until only one remained: a person he didn't know.

Googling him, found it was a junior dev at a FAANG. Oops.

An hour later the contacts repopulated, and all was well. But had to be a white-knuckle time for that poor shmuck.

Re: My $500M Mars rover mistake

#286

Post by euroderf »

atonalfreerider wrote: post #71

Reminds me of the NOAA-N Prime satellite that fell over, because there weren't enough bolts holding it to the test stand. The roots cause, and someone correct me if this is not accurate, was that the x-ray tested bolts to hold it down were so expensive, that they had been "borrowed" to use on another project, and not returned, so that when the time came to flip the satellite into a horizontal position, it fell to the…

And so when "Check for bolts" is added to flip procedure errata/addendum, is light sarcasm called for ?

Re: My $500M Mars rover mistake

#287

Post by 3abiton »

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

Backup saving the day.

Re: My $500M Mars rover mistake

#288

Post by Salgat »

supriyo-biswas wrote: post #234
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

The language is just so anodyne and there’s just that bit of implausible detail in the story (approaching the CEO yourself when you’re the one who fucked up, also how parent claims to be a “top performer” and “I made my company lose millions” at the same time) makes me think this comment was written by an LLM, or at least a fabrication.

That story has been repeated in some form for as long as I was alive.

Re: My $500M Mars rover mistake

#289

Post by euroderf »

natosaichek wrote: post #46
avar wrote: post #12

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis sh…

I've worked on satellites, and yeah - everything is super bespoke, very low quantity, very expensive. There is probably a qualification unit, or a flight spare that may be available for many subsystems, but maybe not. Integration is a long and complicated process. Pulling apart this bot, with however many fasteners, joints, etc, and then reassembling it correctly would be a decidedly non-trivial project that could ea…

Related: NASA's monster rocket costs several times more than SpaceX's monster rocket.

Re: My $500M Mars rover mistake

#290

Post by manfre »

cachvico wrote: post #93
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

I've added a similar safety to every project. It's not perfect, but this last line of defense has saved team members from themselves more than once.

For Django projects, add the below to manage.py:

    env_name = os.environ.get("ENVIRONMENT", "ENVIRONMENT_NOT_SET")
    if env_name in TEST_PROTECTED_ENVIRONMENTS and "test" in sys.argv:
        raise Exception(f"You cannot run tests with ENVIRONMENT={env_name}")
Post reply on HN