That one time I deleted the files on the CEO's laptop and did the backup afterwards - wrong order. Oops.
Never heard anything about it though, so he didn't fully entrust all his data to the intern. Wise decision.
My $500M Mars rover mistake
291–300 of 361 posts
Re: My $500M Mars rover mistake
#292Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…
to be fair, it's a rite of passage to do something like this.
But you should definitely have bought that man a beer :)
Re: My $500M Mars rover mistake
#293Earlier quoted context omitted.
"But it seems like ultimately someone is going to be pretty directly responsible." Or many people, or no one directly. Space missions come with calculated risk. So someone calculates the risk that this critical part brakes is 0.5% and then someone higher up says, that is acceptable and all move on - and then this part indeed brakes and people die. Who is to blame, when the calculation was indeed correct, but 0.5% cha…
> Still, I cannot imagine why anyone thought it was an acceptable risk, to use a 100% oxygen atmosphere with Apollo 1 Especially when, in prior experience there, asbestos had caught fire in the same situation (O2, low pressure).
Wow, this detail I did not know yet. It just was a reckless rush to the moon, no matter the cost at this time. Without the deaths, nothing would have changed probably.
Re: My $500M Mars rover mistake
#294I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move.
However, I don't agree that this is the "real" lesson.
Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them custom test connectors that are incompatible with components that are liable to go up in smoke. That's the lesson. This isn't a little breadboard project they're dealing with, it's a vast project built by countless people in a government agency that has a reputation for formal procedures that are the source of great time, expense, and in some cases ridicule.
The "trust the 28 year old with the $500m robot that can go boom if they slip up" logic seems very peculiar.
Re: My $500M Mars rover mistake
#295Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…
Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…
We had this - 10 years ago. In our case there was a QA environment which was supposed to be used by pushing code up with production configs, then an automated process copied the code to where it actually ran _doing substitutions on the configs to prevent it connecting to the production databases_. However this process was annoyingly slow, and developers had ssh access. So someone (not me) ssh'd in, and sped up their test by connecting the deploy location for their app to git and doing a git pull.
Of course this bypassed the rewrite process, and there was inadequate separation between QA and prod, so now they were connected to the live DB; and then they ran `rake test`...(cue millions of voices suddenly crying out in terror and then being suddenly silenced). The DB was big enough that this process actually took 30 minutes or so and some data was saved by pulling the plug about half-way through.
And _of course_ for maximum blast radius this was one of the apps that was still talking to the old 'monolith' db instead of a split-out microservice, and _of course_ this happened when we'd been complaining to ops that their backups hadn't run for over a week and _of course_ the binlogs we could use to replay the db on top of a backup only went back a week.
I think it was 4 days before the company came back online; we were big enough that this made the news. It was a _herculean_ effort to recover this; some data was restored by going through audit logs, some by restoring wiped blocks on HDs, and so on.
Re: My $500M Mars rover mistake
#296Earlier quoted context omitted.
> I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. Does it inevitably come down to that for someone? I mean even if its a detail that a procedure couldn’t have caught, someone is responsible for forming good procedures. I suppose there could be several factors. But it seems like ultimately someone is going to be pretty directly resp…
"But it seems like ultimately someone is going to be pretty directly responsible." Or many people, or no one directly. Space missions come with calculated risk. So someone calculates the risk that this critical part brakes is 0.5% and then someone higher up says, that is acceptable and all move on - and then this part indeed brakes and people die. Who is to blame, when the calculation was indeed correct, but 0.5% cha…
They had reasons. See: https://en.wikipedia.org/wiki/Apollo_1#Choice_of_pure_oxygen...
Even after the fire, the Apollo spacecraft still used 100% oxygen when in space. The cabin was 60% oxygen / 40% nitrogen at 14.7 psi at launch, reducing to 5 psi on ascent by venting, with the nitrogen then being purged and replaced with 100% oxygen.
> See Murpheys Law...
Indeed. I hope that was a joke.
Re: My $500M Mars rover mistake
#297I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…
That reminds me of a fiction-quote by one of my favorite authors, where a welding-instructor has just finished sharing a (somewhat literal) post-mortem anecdote of falsified safety inspections. > He gathered his breath. “This is the most important thing I will ever say to you. The human mind is the ultimate testing device. You can take all the notes you want on the technical data, anything you forget you can look up…
Aside: I recognised 'quaddies' from your quote ... there was some very distinctive cover art on the Analog magazine for that story: https://www.abebooks.co.uk/Analog-Science-Fact-Fiction-Febru...
Re: My $500M Mars rover mistake
#298Earlier quoted context omitted.
Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)
Everybody replying to you that this is fragile is missing the point. This kind of code isn't the first line of defense—it's the last.
And when your last line of defense fires... you don't just breath a sigh of relief that the system is robust. You also must dig in to how to catch it sooner in your previous lines.
For instance, test code shouldn't have access to production DB passwords. Maybe that means a slightly less convenient login for the dev to get to production, but it's worth it.
Re: My $500M Mars rover mistake
#299I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them cus…
That was my first thought as well.
On the other hand, it's hard to make these kinds of judgment calls when you're talking about a one-off piece of equipment that's only going to go through this particular testing cycle a single time.
In computing, there are a lot of similar "one-off" operations -- something you to do to the prod database or router config a single time as part of an upgrade or migration.
Sometimes building a safeguard is more effort than just paying attention in the first place. And while we don't always perfectly pay attention, we also don't always perfectly build safeguards, and wind up making similar mistakes because we're trusting the faulty safeguard.
In circumstances like the one in the story, the best approach might almost be the hardware equivalent of pair programming -- the author should have had a partner solely responsible for verifying everything he did was correct. (Not just an assistant like Mary who's helping, where they're splitting responsibilities -- no, somebody whose sole job is to follow along and verify.)
Re: My $500M Mars rover mistake
#300Earlier quoted context omitted.
Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update…
Lesson is never attempt to do anything on a Friday afternoon that will take far more time for recovery.
Or is the lesson to _always_ attempt such critical changes on a Friday? After all, in this instance the client didn't notice any problems, apparently because they were already off to their weekend.
For me personally the much bigger issue would be harming the client, their business or our relationship. Doing a few hours of overtime to fix my mistakes would probably only feel as well deserved punishment...