Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

161–170 of 361 posts

Re: My $500M Mars rover mistake

#161

Post by quacksilver »

mk_stjames wrote: post #130

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

I'm surprised that group therapy for engineers doesn't exist or maybe I just can't find it. I don't work in anything high stakes myself though have often been an ear to those who do in aviation or rail.

I believe that it can be quite hard sometimes if you have empathy and don't take the "Once the rockets are up, who cares where they come down? That's not my department" approach."

Perhaps some peer support group (possibly facilitated) for people that build safety critical systems or deal with the fallout. Not all companies will provide good counseling etc.

Perhaps the engineering boards / chartered engineer organizations should provide this and fund it from their membership fee, though that would probably scare people off going to the service as they could be afraid of losing their stamp / chartered engineer status / license.

Perhaps this would be dealt with in the past by getting drunk with colleagues in the pub, though alcoholism (or being impaired at work the next morning) is bad and pubs etc. are less popular now.

Re: My $500M Mars rover mistake

#162

Post by stylepoints »

cachvico wrote: post #93
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

#metoo

Re: My $500M Mars rover mistake

#163

Post by refactor_master »

From reading the developer anecdotes in here I think it’s worth mentioning that if just one person can bring down the whole enterprise, a hacker only needs a point of entry to do the same.

For our databases we have separate credentials, compartmentalized access and disallowed “dangerous” commands. This now seems like obviousness, but we only got this years in. Thankfully, no (major) incidents have occurred to this date.

Re: My $500M Mars rover mistake

#164

Post by [deleted] »

Terr_ wrote: post #153
mk_stjames wrote: post #130

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

That reminds me of a fiction-quote by one of my favorite authors, where a welding-instructor has just finished sharing a (somewhat literal) post-mortem anecdote of falsified safety inspections. > He gathered his breath. “This is the most important thing I will ever say to you. The human mind is the ultimate testing device. You can take all the notes you want on the technical data, anything you forget you can look up…

[deleted]

Re: My $500M Mars rover mistake

#165

Post by ChristianGeek »

nonethewiser wrote: post #158
mk_stjames wrote: post #130

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

> I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. Does it inevitably come down to that for someone? I mean even if its a detail that a procedure couldn’t have caught, someone is responsible for forming good procedures. I suppose there could be several factors. But it seems like ultimately someone is going to be pretty directly resp…

I’d be interested in talking to the team at Boeing.

Re: My $500M Mars rover mistake

#166

Post by quasse »

quacksilver wrote: post #161
mk_stjames wrote: post #130

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure inv…

I'm surprised that group therapy for engineers doesn't exist or maybe I just can't find it. I don't work in anything high stakes myself though have often been an ear to those who do in aviation or rail. I believe that it can be quite hard sometimes if you have empathy and don't take the "Once the rockets are up, who cares where they come down? That's not my department" approach." Perhaps some peer support group (poss…

Profession induced trauma is finally getting taken seriously in the medical setting, another high stakes field.

Group therapy for doctors and nurses is finally becoming a thing, but unfortunately it is completely dependent on being employed by an organization that cares about it.

Re: My $500M Mars rover mistake

#167

Post by 0xDEAFBEAD »

yafbum wrote: post #140
askvictor wrote: post #109

Earlier quoted context omitted.

One place I worked (some 20 years ago) had a policy that any time you run a sudo command, another person has to check the command before you hit enter. Could apply the same kind of policy/convention for anything in production.

I'm not sure this doesn't just lead to blind rubber-stamping unless this is done very very rarely

The trick is to have good access controls so confirmations happen often enough to be useful, but not so often to be rubber-stamped

Re: My $500M Mars rover mistake

#168

Post by throwawayosiu1 »

I'll add my story here for posterity:

My first job out of university, I was working for a content marketing startup who's tech stack involved PHP and PerconaDB (MySQL). I was relatively inexperienced with PHP but had the false confidence of a new grad (didn't get a job for 6 months after graduating - so I was desperate to impress).

I was tasked with updating some feature flags that would turn on a new feature for all clients, except for those that explicitly wanted it off. These flags were stored in the database as integers (specifically values 4 and 5) in an array (as a string).

I decided to use the PHP function (array_reverse)[https://www.php.net/manual/en/function.array-reverse.php] to achieve the necessary goal. However, what I didn't know (and didn't read up on the documentation) is that, without the 2nd argument, it only reversed the values not the keys. This corrupted the database with the exact opposite of what was needed (somehow this went through QA just fine).

I found out about this hours later (used to commute ~3 hrs each way) and by that time, the senior leadership was involved (small startup). It was an easy fix - just a reverse script - but it highlighted many issues (QA, DB Backups not working etc.)

I distinctly remember (and appreciate) that the lead architect came up to me the next day and told me that it was rite of passage of working with PHP - a mistake that he too had made early in his career.

I ended up being fired (grew as an engineer and was better off for it) but in that moment and weeks after it, it definitely demoralized me.

Re: My $500M Mars rover mistake

#169

Post by Feathercrown »

markmark wrote: post #160
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each…

Isn't that almost exactly what happened at github too?

Re: My $500M Mars rover mistake

#170

Post by notRobot »

throwawayosiu1 wrote: post #168

I'll add my story here for posterity: My first job out of university, I was working for a content marketing startup who's tech stack involved PHP and PerconaDB (MySQL). I was relatively inexperienced with PHP but had the false confidence of a new grad (didn't get a job for 6 months after graduating - so I was desperate to impress). I was tasked with updating some feature flags that would turn on a new feature for all…

They fired you for that array_reverse mistake?
Post reply on HN