Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

251–260 of 361 posts

Re: My $500M Mars rover mistake

#251

Post by augustk »

cachvico wrote: post #93
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

And then it turns out that the order of the parameters was mixed up... just kidding.

Re: My $500M Mars rover mistake

#252

Post by MagicMoonlight »

irjustin wrote: post #16

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of th…

The fact that they were still being made to work after completing a 12 hour shift (which is already too long to be safe) means this was a process error

Re: My $500M Mars rover mistake

#253

Post by jve »

augustk wrote: post #251
cachvico wrote: post #93

Earlier quoted context omitted.

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

And then it turns out that the order of the parameters was mixed up... just kidding.

Just yesterday, I did C# Regex.Match with a supersimple regex: ^\d+ And it seemed not to work. I asked ChatGTP and he noted that I had subtle mistake: the parameters were other way around... :facepalm:

Re: My $500M Mars rover mistake

#254

Post by creshal »

mango7283 wrote: post #243
hutzlibu wrote: post #240

Earlier quoted context omitted.

The suspicious part for me would be the CEO laughing like it was nothing. Also yes, one would expect it goes the other way around, when you messed up big, someone will come to you. But the world is big and maybe it happened like this.

Surely this is a variation of this anecdote attributed to IBM's Watson https://news.ycombinator.com/item?id=13419313 "> A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to ha…

A variant is in From the Earth to the Moon, where a junior engineer at Grumman confesses messing up vital calculations for the Lunar Lander to his boss, and finishes with "So… I guess I'll go clean out my desk." "What for?" "I figure you're gonna fire me now."

The boss's response makes a lot more sense than the usual fluff, though: "If I fire you now, the next guy to make a mistake won't admit it and we won't find out about it until it's too late."

Re: My $500M Mars rover mistake

#255

Post by creshal »

thfuran wrote: post #204
bboygravity wrote: post #203

Earlier quoted context omitted.

You worried about that? I'm a frequent flyer and I got a feeling that most airline ticket booking pages are broken in some way more than half the time. Maybe not often broken to the point that they're blank, but definitely broken to the point that booking a ticket isn't possible (I prefer blank, so that I don't waste like 30 minutes on not being able to book a ticket). Also most of the internet seems often broken. Oh…

What always got me is that, for at least the first several years, Google couldn't get their store page to handle the load when they were releasing a new phone and it'd be crapping out for days.

Steam still craps out during large sales. I really wonder how Valve calculates that it's fine to keep losing out on (cumulatively) hours of sales each time.

Re: My $500M Mars rover mistake

#256

Post by nojvek »

I’m glad they didn’t fire him.

I’m a firm believer that despite all the short comings of US, what makes it great is there are millions of engineers and scientists working to push the frontier of what is possible, and trillions of dollars in economy to fund that into reality.

NASA is truly an inspiration.

And also the private aerospace industry - SpaceX, ULA, Boeing, Lockheed Martin, Blue Origin, Planet Labs etc.

No other country has that.

Re: My $500M Mars rover mistake

#257

Post by belter »

supriyo-biswas wrote: post #234
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

The language is just so anodyne and there’s just that bit of implausible detail in the story (approaching the CEO yourself when you’re the one who fucked up, also how parent claims to be a “top performer” and “I made my company lose millions” at the same time) makes me think this comment was written by an LLM, or at least a fabrication.

A CEO at the office and not in the golf course...that gives it away....

Re: My $500M Mars rover mistake

#258

Post by Thorrez »

supriyo-biswas wrote: post #234
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

The language is just so anodyne and there’s just that bit of implausible detail in the story (approaching the CEO yourself when you’re the one who fucked up, also how parent claims to be a “top performer” and “I made my company lose millions” at the same time) makes me think this comment was written by an LLM, or at least a fabrication.

Has it been edited? I don't see "top performer".

Re: My $500M Mars rover mistake

#259

Post by prox »

dotsam wrote: post #231
markmark wrote: post #160

Earlier quoted context omitted.

At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each…

> logged into his dev db and the prod db at the same time in different windows I am very worried about doing the wrong thing in the wrong terminal, so for some machines I colour-code my ssh windows, red for prod, yellow for staging and green for dev. e.g. in my ~/.bashrc I have: echo -ne '\e]11;#907800\a' #yellow background

This is a good idea, especially when tired, end of day, crunch time work is happening !
Post reply on HN