Live data from Hacker News

We deleted the production database by accident

keepthescore.co

281–290 of 456 posts

Re: We deleted the production database by accident

#281
post #268
post #267

Earlier quoted context omitted.

You should have zero fear instilled when pressing any button. The system or process has failed if a single person pressing a button can bring something down unintended. Fix the system/process, don’t “instill fear” onto the person, it’s toxic, plus now you have to make sure any new person on boarded has “fear instilled”, and that’s just hugely unproductive

That’s precisely my point. A lot of people have no fear because they’re complacent or ignorant rather than the button is well engineered. But to get there you need to fear the bad outcomes.

I’m sorry, but this really hasn’t been my experience at all in web technology or managing on-prem systems either.

I used to be extremely fearful of making mistakes, and used to work in a zero-tolerance fear culture. My experience and the experience of my teammates on the DevOps team? We did everything manually and slowly because we couldn’t see past our own fear to think creatively on how to automate away errors. And yes, we still made errors doing everything carefully, with a second set of eyes, late night deployments, backups taken, etc.

Once our leadership at the time changed to espouse a culture of learning from a mistake and not being afraid to make one as long as you can recover and improve something, we reduced a lot of risk and actually automated away a lot of errors we typically made which were caused initially by fear culture.

Just my two cents.

Re: We deleted the production database by accident

#282

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

Thanks

Re: We deleted the production database by accident

#283
The maturity of the article is laughable. I'm sure my age is the same as the people who wrote it, but this is unacceptable: dropping databases in prod is a serious issue, not a joke. I think the culture of the company is toxic and not professional at any level. #change-my-mind

Re: We deleted the production database by accident

#284

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

I have no employees. I only have myself to blame

Re: We deleted the production database by accident

#285
post #281
post #268

Earlier quoted context omitted.

That’s precisely my point. A lot of people have no fear because they’re complacent or ignorant rather than the button is well engineered. But to get there you need to fear the bad outcomes.

I’m sorry, but this really hasn’t been my experience at all in web technology or managing on-prem systems either. I used to be extremely fearful of making mistakes, and used to work in a zero-tolerance fear culture. My experience and the experience of my teammates on the DevOps team? We did everything manually and slowly because we couldn’t see past our own fear to think creatively on how to automate away errors. And…

I’m not talking about fear culture. It’s a more personal thing. Call it risk management culture if that helps which is the inverse.

Manual is wrong for a start. That would increase the probability of making a mistake and thus increase risk for example. The mitigations are automation, review and testing.

I agree with you. Perhaps fear was the wrong term. I treat it as my personal guide to how uneasy i feel about something on the quality front.

Re: We deleted the production database by accident

#286

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

The whole piece has a slightly annoying flippant tone to it. We were drunk! Computers just.. do this stuff sometimes! Better to sound contrite and boring in such a situation IMO. Also I agree with other comments: doing some work after a glass or two should be fine because you should have other defences in place. “Not being drunk” shouldn’t be the only protection you have against disaster.

Yeah, I agree I'm being slightly flippant.

But it's just a side-project and I will continue late night coding with a glass of wine. I find it hugely enjoyable.

I would have a different mind-set if I was writing software for power stations as a professional.

Re: We deleted the production database by accident

#287

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

When I was getting my pilots license I used to read accident reports from Canada's Transportation Safety Board [1]. I'm sure the NTSB (America's version) has similar calibre reports [2].

There is also Cockpit Resource Management [3] which addresses the human factor in great detail (how people work with each other, and how prepared are people).

In general what you learn from reading these things is that its rarely one big error or issue - but many small things leading to the failure event.

1 - https://www.tsb.gc.ca/eng/rapports-reports/aviation/index.ht...

2 - https://www.ntsb.gov/investigations/AccidentReports/Pages/Ac...

3 - https://en.wikipedia.org/wiki/Crew_resource_management

Re: We deleted the production database by accident

#288

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

2 cents his hosts file points localhost to the prod db IP

My guess would be something like PgBouncer. Someone may have installed it to the production server at some point in the past.

Re: We deleted the production database by accident

#289
post #238

Earlier quoted context omitted.

And you didn’t even bother to do a query of the actual maximum length value of the columns you were mutating? Or at least query and see the text in there? Basically you just blindly ran the migration on the data and checked if it didn’t fail? The lesson here is not about cleverness unfortunately.

The comment clearly states that they did.

If they did, they would have noticed that the columns were empty (because they were wiped clean for PI data).

The parent is either misrepresenting the situation or they didn’t do what they say they did.

Also in any production setup, before the migration in the same transaction you would have something along the lines of “check if the column size is larger than and then abort”, because you never know when that can be added while working on the database.

Re: We deleted the production database by accident

#290

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

This is also one of the core tenets of SRE. The chapter on blameless postmortems is quite nice: https://landing.google.com/sre/sre-book/chapters/postmortem-...
Post reply on HN