Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

111–120 of 361 posts

Re: My $500M Mars rover mistake

#111

Post by toasted-subs »

cachvico wrote: post #93
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

Yup, I have 3 prompts if you want to wipe anything.

One of the reasons I put interactions between databases behind a cli.

Re: My $500M Mars rover mistake

#112

Post by stickfigure »

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

I have a rule when working on production databases: Always `start transaction` before doing any kind of update - and pay close attention to the # of rows affected.

Re: My $500M Mars rover mistake

#113

Post by [deleted] »

askvictor wrote: post #109
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

One place I worked (some 20 years ago) had a policy that any time you run a sudo command, another person has to check the command before you hit enter. Could apply the same kind of policy/convention for anything in production.

[deleted]

Re: My $500M Mars rover mistake

#115

Post by amluto »

maxerickson wrote: post #95
lisper wrote: post #85

Earlier quoted context omitted.

Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move i…

So how do you prevent them from grabbing the wrong break out box? Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor? Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?

I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack.

So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors.

All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.

Re: My $500M Mars rover mistake

#116

Post by vlovich123 »

galangalalgol wrote: post #100
vlovich123 wrote: post #84

Earlier quoted context omitted.

They do know roughly how long it takes to take care of a patient & should be set up with overlapping shifts and to be winding down towards a normal shift (i.e. no new patients) so that there's no handoff of a single patient but no one is working long hours. Some patients might take longer than a single shift, but handoff is inevitable at some point. You can improve your handoff processes but you can't improve the dec…

Is this maybe one of those "if something is hard, do it more often" things?

Maybe but I won't claim to know how to quantify things to evaluate proposals. I do know that even in tech with low stakes, hand off is a problem. I recall hearing teams trying to do 3 on-call teams in 3 different timezones and the team requested to scale back to 2 with longer hours because of the handoff problem (& these hand-offs were occurring daily).

Re: My $500M Mars rover mistake

#117

Post by devsda »

cachvico wrote: post #93
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

Ours are not named with a common identifier and this also needs constant effort to maintain while refactoring and there's still scope for a mistake.

*ideally* devs should not have prod access or their credentials should only have limited access without permissions for destructive actions like drop/truncate etc.

But in reality, there's always that one helpful dba/dev who shares admin credentials for a quick prod fix with someone and then those credentials end up in a wiki somewhere as part of an SOP.

Re: My $500M Mars rover mistake

#118

Post by csomar »

cachvico wrote: post #93
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

This is bad because if someone forgot to add prod or for whatever reason the code executed beyond the panic, you’ll wipe out the db.

There is no code that will protect your db/data. Only replication to a read-only storage will help in such situations.

Re: My $500M Mars rover mistake

#119

Post by totallywrong »

jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

As a young consultant, I was once one Enter away from causing a disaster, but something stopped me. I still shudder even though it didn't actually happen. Nothing of the sort in many years since, so a great lesson in retrospect I guess.

Re: My $500M Mars rover mistake

#120

Post by aunwick »

Congratulations. 102 circuits would have taught this. But some short sighted bean counters merged EE & CE so you didn't get the opportunity. I suggest you ask for a tuition refund as they failed to educate you as they promised.
Post reply on HN