Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My $500M Mars rover mistake

chrislewicki.com

Post reply on HN
Print view View original

231–240 of 361 posts

Re: My $500M Mars rover mistake

#231

Post by dotsam »

markmark wrote: post #160
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each…

> logged into his dev db and the prod db at the same time in different windows

I am very worried about doing the wrong thing in the wrong terminal, so for some machines I colour-code my ssh windows, red for prod, yellow for staging and green for dev. e.g. in my ~/.bashrc I have: echo -ne '\e]11;#907800\a' #yellow background

Re: My $500M Mars rover mistake

#232

Post by danw1979 »

blauditore wrote: post #215

The fact this can so easily happen shows a lack of safety mechanisms. All the software-related stories in this comment section go in a similar direction: They could have been prevented by simple safety nets. If you accidentally wipe a production database, then it was likely too easy to do so. Don't blame humans for occasional mistakes, it won't stop them from happening.

The idea that one can learn to not make errors is toxic. To err is human. Sure, you can get more reliable at something, but everyone - even the most experienced - will fry the rover at some point.

Re: My $500M Mars rover mistake

#233

Post by davedx »

Cthulhu_ wrote: post #230
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

I think the loss may not have been as much as you think; sure, nobody could buy tickets for a few hours, so theoretically the company lost millions of revenue during that time. But that assumes people wouldn't just try again later. Downtime does not, in practice, translate to losses I think. I mean look at Twitter, which was famously down all the time back when it first launched due to it popularity and architecture.…

Even with HFT you’d have to have more than 50% of your trades go against you to lose any money, and you’ll probably have hedges, and losing some % of money will be within normal operation parameters. Shit happens! Links go down, hardware fails, bugs slip through no matter how diligent you are. (No I’m not looking to be hired by any HFT shops)

Re: My $500M Mars rover mistake

#234

Post by supriyo-biswas »

drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

The language is just so anodyne and there’s just that bit of implausible detail in the story (approaching the CEO yourself when you’re the one who fucked up, also how parent claims to be a “top performer” and “I made my company lose millions” at the same time) makes me think this comment was written by an LLM, or at least a fabrication.

Re: My $500M Mars rover mistake

#235

Post by holografix »

drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

Wow chatgpt is actually getting worse.

Re: My $500M Mars rover mistake

#236

Post by masklinn »

devsda wrote: post #117
cachvico wrote: post #93

Earlier quoted context omitted.

Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)

Ours are not named with a common identifier and this also needs constant effort to maintain while refactoring and there's still scope for a mistake. *ideally* devs should not have prod access or their credentials should only have limited access without permissions for destructive actions like drop/truncate etc. But in reality, there's always that one helpful dba/dev who shares admin credentials for a quick prod fix w…

That‘s why you do credentialing via ssh keys, and keys are explained and map to a user, and non-dba keys should expire.

If you need access for a quick prod fix, your key gets added to the machine with that explanation and a week (or lees) lifetime.

Re: My $500M Mars rover mistake

#237

Post by xattt »

donatj wrote: post #157
jay-barronville wrote: post #67

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had…

Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update…

Lesson is never attempt to do anything on a Friday afternoon that will take far more time for recovery.

Re: My $500M Mars rover mistake

#238

Post by masklinn »

norman784 wrote: post #211
cdogl wrote: post #92

Earlier quoted context omitted.

Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or t…

Our test suite expects that the database name has a `_test` suffix, so you can't run the tests even locally without the suffix.

Our test harness takes an optional template as input and immediately copies it.

It’s useful to distribute the test anyway, especially for non-transactional tests.

If the database initialisation is costly that’s useful even if tests run on empty, as copying a database from a template is much faster than creating one DDL by DDL, for postgres at least.

Re: My $500M Mars rover mistake

#239

Post by hnarayanan »

skrebbel wrote: post #184
Scubabear68 wrote: post #89

Earlier quoted context omitted.

At a major brokerage firm I accidentally hit prod with a testing script that did several millions of dollars of fake FX test trades. The first thing mentioned in the post mortem call was “No one is going to blame the guy who did those trades. It was an honest mistake. What we are going by to do is discuss why a developer can hit the production trading API without any authentication at all”.

Were the trades any good though?

Plot twist: It made so much money that that’s now their strategy.

Re: My $500M Mars rover mistake

#240

Post by hutzlibu »

supriyo-biswas wrote: post #234
drra wrote: post #199

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the cr…

The language is just so anodyne and there’s just that bit of implausible detail in the story (approaching the CEO yourself when you’re the one who fucked up, also how parent claims to be a “top performer” and “I made my company lose millions” at the same time) makes me think this comment was written by an LLM, or at least a fabrication.

The suspicious part for me would be the CEO laughing like it was nothing. Also yes, one would expect it goes the other way around, when you messed up big, someone will come to you. But the world is big and maybe it happened like this.
Post reply on HN