Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

211–220 of 361 posts

Re: My $500M Mars rover mistake

#211

Post by norman784 »

cdogl wrote: post #92
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Our test suite expects that the database name has a `_test` suffix, so you can't run the tests even locally without the suffix.

Re: My $500M Mars rover mistake

#212

Post by rahimnathwani »

drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

This reminds me of that old joke that ends "Why would I fire you? We just spent millions training you!".

People who take on high risk projects are underappreciated. But many managers prefer employees who can reliably deliver zero value, than those with positive expected value but non-zero variance.

Re: My $500M Mars rover mistake

#213

Post by globalise83 »

fullspectrumdev wrote: post #181
throwawayosiu1 wrote: post #171

Earlier quoted context omitted.

Yep, I was put on PIP with impossible success criteria (no issues raised in PRs by senior engineers and no issues in code deployed to production - even if it was reviewed by senior engineers & QA) and fired (for failing that criteria) in 2 weeks. I worked there for ~8 months in total.

The impossible PIP trick for dismissal is something I’d love to see get eventually legally obliterated.

Constructive dismissal is illegal in many countries. It's the choice of the people which system they want to work in.

Re: My $500M Mars rover mistake

#214

Post by mr_mitm »

rahimnathwani wrote: post #212
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

This reminds me of that old joke that ends "Why would I fire you? We just spent millions training you!". People who take on high risk projects are underappreciated. But many managers prefer employees who can reliably deliver zero value, than those with positive expected value but non-zero variance.

That story sounds so much like that joke that I'm wondering if there is some urban legend thing going on here.

Re: My $500M Mars rover mistake

#215

Post by blauditore »

The fact this can so easily happen shows a lack of safety mechanisms. All the software-related stories in this comment section go in a similar direction: They could have been prevented by simple safety nets. If you accidentally wipe a production database, then it was likely too easy to do so.

Don't blame humans for occasional mistakes, it won't stop them from happening.

Re: My $500M Mars rover mistake

#216

Post by tempestn »

sopooneo wrote: post #149
maxerickson wrote: post #131

Earlier quoted context omitted.

It started up fine. The multimeter was connecting up the telemetry, so they weren't getting any information from it until they restored that circuit. The power absolutely did feed into that circuit, they were trying to decide if it would have damaged it (but a motor driver is going to be able to handle power coming from the motor, so they decided that it probably didn't damage it).

That is my reading too. But why was the multimeter connecting up telemetry? That seems very strange to me.

I assume it was wired up in series to measure current.

Re: My $500M Mars rover mistake

#217

Post by prmoustache »

curiousObject wrote: post #104

The story is compellingly written, but I thought it was also confusing. It sounds as if this team made several mistakes, not just one mistake. It's also not clear if the result of these mistakes was that there might be real damage to the spacecraft, or if the result was just wasted time and hours of confusion about why the spacecraft wouldn't start up. The first mistake is they didn't realize that the multimeter was…

You forgot: the telemetry guy (Leo) didn't mention they had lost Telemetry before the storyteller told him he did a mistake. I mean shouldn't they have cancelled all testing until they have it back?

Re: My $500M Mars rover mistake

#218

Post by bambax »

Great write-up. My two favorite quotes:

> I had learned from countless experiences in this and other projects that bad news doesn’t get better with age

That's so true! We tend to sit on bad news and hope that somehow time will blunt them; but if anything the opposite happens.

And

> I still remember the shock when Project Manager Pete delivered the decision and the follow-on news: ‘These tests will continue. And Chris [the author] will continue to lead them as we have paid for his education. He’s the last person on Earth who would make this mistake again.’

We sometimes think people who made one mistake will make another one, and it's better to go with the person who doesn't make mistakes. But that's not the correct approach. People who don't make mistakes are often people who don't do anything.

Re: My $500M Mars rover mistake

#219

Post by slowmotiony »

drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

There must be more people like you in those major airlines, as those sites go down all the damn time. 6 hours..??? The lufthansa desktop site didn't allow anyone book anything for like 3 weeks straight, you had to use the app instead.

Re: My $500M Mars rover mistake

#220

Post by BiteCode_dev »

bboygravity wrote: post #203
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

You worried about that? I'm a frequent flyer and I got a feeling that most airline ticket booking pages are broken in some way more than half the time. Maybe not often broken to the point that they're blank, but definitely broken to the point that booking a ticket isn't possible (I prefer blank, so that I don't waste like 30 minutes on not being able to book a ticket). Also most of the internet seems often broken. Oh…

Hell, I used to worry about down time for my tiny blog. Didn't want to let down my readers.

Everything can be a guilt trip if you try hard enough.

Then I met a guy, now a good friend, that made me do my first "pull the plug migration" on his most important website. He lived on this.

I looked at the site going down, horified. He mocked me, then proceeded with the udate. It didn't work. The site stayed offline for hours.

Then it worked again. And nobody cared. It had zero consequences on traffic.

User were pissed off for a few hours, and life goes on.

Post reply on HN