We deleted the production database by accident • Hacker News Board

Re: We deleted the production database by accident

#411

Post by cutemonster » Sun, Oct 18, 2020, 5:33 PM UTC

xtracto wrote: post #403

rocqua wrote: post #257

Earlier quoted context omitted.

Misclicking is a tiny forgivable mistake. Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

This. The person that erased the database in my case came forward to me as soon as we realized what had happened. At that moment I was very happy it was an "inside job", it meant I could discard hacking. As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.

How do you prevent forwarding ports? Then one needs to disable ssh access?

Nice that you were a person he felt ok with sharing the mistake with, I suppose that's an important part of being head of eng.

Re: We deleted the production database by accident

#412

Post by neillyons » Sun, Oct 18, 2020, 5:35 PM UTC

Jugurtha wrote: post #333

neillyons wrote: post #265

Earlier quoted context omitted.

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

"The Checklist Manifesto", by Atul Gawande, dives into how they looked at other sectors such as aviation to improve healthcare systems, reduce infections, etc. Interesting book.

Just bought the audiobook. About to give it a listen now. Thanks.

Re: We deleted the production database by accident

#413

Post by cutemonster » Sun, Oct 18, 2020, 5:36 PM UTC

ojnabieoot wrote: post #349

rocqua wrote: post #257

Earlier quoted context omitted.

Misclicking is a tiny forgivable mistake. Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

It sounds like whoever did it might not even be aware they were responsible: > in the dropdown menu of the MongoDB browser, exit & drop database were next to each other So maybe they signed off for the night without realizing anything was wrong.

MongoDB can sell an enterprise version with the buttons further apart

Re: We deleted the production database by accident

#414

Post by xupybd » Sun, Oct 18, 2020, 5:41 PM UTC

I don't understand how this happened if localhost is hard coded and the password is different. I don't think they fully understand why this happened. At least enough to prevent it from happening again.

Re: We deleted the production database by accident

#415

Post by cutemonster » Sun, Oct 18, 2020, 5:42 PM UTC

StavrosK wrote: post #73

muststopmyths wrote: post #7

>Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. We’re too tired to figure it out right now. The gremlins won this time. Obviously, somehow the script ran on the database host. some practices I've followed in the past to keep this kind of thing from happening: * A script that deletes all the data can never be deployed to production. * scrip…

Someone SSHed to production and forwarded the database port to the local machine to run a report, then forgot about the connection and ran the deletion script locally.

That has happened? Or was it a thought about what could have happened elsewhere?

Re: We deleted the production database by accident

#416

Post by [deleted] » Sun, Oct 18, 2020, 6:15 PM UTC

JackFr wrote: post #386

janoc wrote: post #366

Earlier quoted context omitted.

There is almost never a single cause. If a single cause can trigger a disaster, then there is another cause by definition - poor system design. E.g. in the article's case it is clear that there is some sort of procedural deficiency there that allows the configuration variables to be set wrong and thus cause a connection to the wrong database. Another one is that the function that has directly caused the data loss DOE…

This should be a teachable moment with respect to their culture. Throwing up their hands without an understanding of what happened is unacceptable — if something that is believed impossible happens, it is important to know where your mental model failed. Otherwise you may make things worse by ‘remediating’ the wrong thing. And while this sounds overly simplistic the simplest way this could have been avoided is enforc…

[deleted]

Re: We deleted the production database by accident

#417

Post by rawgabbit » Sun, Oct 18, 2020, 6:18 PM UTC

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

For me, this is an example of DevOps being carried too far.

What is to stop developers for checking into Github "drop database; drop table; alter index; create table; create database; alter permission;"? They are automating environment builds and so that is more efficient right? In my career, I have seen a Fortune 100 company's core system down and out for a week because of hubris like this. In large companies, data flows downstream from a core system. When you have to restore from backup, that cascades into restores in all the child systems.

Similarly, I once had to convince a Microsoft Evangelist who was hired into my company, not to redeploy our production database, every-time we had a production deployment. He was a pure developer and did not see any problems of dropping the database, recreating the database, and re-inserting all the data. I argued that a) this would take 10+ hours b) the production database has data going back many years and that the schema/keys/rules/triggers have evolved during that time-- meaning that many of the inserts would fail because they didn't meet the current schema. He was unconvinced but luckily my bosses overruled him.

My bosses were business types and understood accounting. In accounting, once you "post" a transaction to the ledger that becomes permanent. If you need to correct that transaction, then you create a new one that "credits" or corrects the entry. You don't take out the eraser.

Re: We deleted the production database by accident

#418

Post by ThrowawayR2 » Sun, Oct 18, 2020, 6:20 PM UTC

henearkr wrote: post #147

bstar77 wrote: post #134

Earlier quoted context omitted.

I am not a drinker myself (drink 1-3 times a year), but in the past I have coded while slightly buzzed on a few occasions. I could not believe the level of focus I had. I never investigated it further, but I'm pretty sure the effects of alcohol on our coding abilities is not nearly as bad as it affects our motor skills. Imo, fatigue is far worse.

When I was not yet a teetotaler, each time I was hitting my maths textbooks after a few drinks, I could not believe my level of focus, and everything was clear and obvious. Textbook pages were flying at a speed never seen. Of course the next day, when re-reading the same pages, I was always discovering that the previous day I had everything wrong, nothing was obvious, and all my reasoning when with alcohol was false…

Not a great idea for studying anyway because of https://en.wikipedia.org/wiki/State-dependent_memory

In short, ability to recall memories is at least in part dependent on being in a similar state to the time when memories are formed, e.g. something learned while being intoxicated will be more easily recalled only when intoxicated again.

Re: We deleted the production database by accident

#419

Post by StavrosK » Sun, Oct 18, 2020, 6:27 PM UTC

cutemonster wrote: post #415

StavrosK wrote: post #73

Earlier quoted context omitted.

Someone SSHed to production and forwarded the database port to the local machine to run a report, then forgot about the connection and ran the deletion script locally.

That has happened? Or was it a thought about what could have happened elsewhere?

Oh, no, that's my guess as to what happened here.

Re: We deleted the production database by accident

#420

Post by suzakus » Sun, Oct 18, 2020, 7:02 PM UTC

ozim wrote: post #279

suzakus wrote: post #246

Earlier quoted context omitted.

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

Parent is not advocating about going as strict with procedures as operating an airplane. Post is saying about "a bit of aviation safety culture" then it highlights a specific part that would be useful. Safety culture element highlighted is: not blaming a single person but finding out how to prevent accident that happened from happening again. Which is reasonable because you don't want to impose some strict rules that…

Ah, I misread. That's what I get for commenting late at night :(

Thanks for clarifying!