We deleted the production database by accident • Hacker News Board

Re: We deleted the production database by accident

#441

Post by amingilani » Mon, Oct 19, 2020, 1:29 PM UTC

netheril96 wrote: post #127

amingilani wrote: post #47

Happens to all of us. Once I required logs from the server. The log file was a few gigs and still in use. so I carefully duplicated it, grepped just the lines I needed into another file and downloaded the smaller file. During this operation, the server ran out of memory—presumably because of all the files I'd created—and before I know it I'd managed to crash 3 services and corrupted the database—which was also on thi…

Why does the DB get corrupted? Does ACID mean anything these days?

It was an older MongoDB in my case. :)

Re: We deleted the production database by accident

#442

Post by emodendroket » Mon, Oct 19, 2020, 2:34 PM UTC

tcbasche wrote: post #428

Aeolun wrote: post #291

Earlier quoted context omitted.

> taking responsibility for their own incompetence. I think they’re doing that with this post? At least I find it hard to imagine myself writing down that I’d drunk a few glasses of wine and dropped the production database. You cannot expect all engineers to be fully versed in the vagarities of database administration. Especially if they’re the only ones working on something.

Not really, they blamed 'complexity' and 'computer gremlins' rather than admitting that, perhaps, they made a shitty mistake. > It’s a function that deletes the local database and creates all the required tables from scratch Why would anyone have this? It's just dumb and embarrassing

If you don't put your database in a known state for tests they're not repeatable

Re: We deleted the production database by accident

#443

Post by jsmith45 » Mon, Oct 19, 2020, 2:51 PM UTC

neillyons wrote: post #265

t0mas88 wrote: post #243

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

Not sure about books, but the NTSB generally seems to adopt the philosophy of not trying to assign blame, but instead to figure out what happened, and try to determine what can be changed to prevent this same issue from happening again.

Of course trying to assign blame is human nature, so the reports are not always completely neutral. When I read the actual NTSB report for Sullenburger's "Miracle on the Hudson", I was forced to conclude that while there were some things that the pilots could in theory have done better, given the pilots training and documented procedures, they honestly did better than could reasonably be expected. I am nearly certain that some of the wording in the report was carefully chosen to lead one to this conclusion, despite still pointing out the places where the pilots actions were suboptimal (and thus appearing facially neutral).

The "what can we do to avoid this ever happing again?" attitude applies to real air transit accident reports. Sadly many general aviation accident reports really do just become "pilot error".

Re: We deleted the production database by accident

#444

Post by fireant » Mon, Oct 19, 2020, 5:29 PM UTC

jonplackett wrote: post #425

janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

It’s interesting that this is the exact opposite of how we think about crime and punishment. All criminals are like the pilot, just the person who did the action. But the reasons for them becoming criminals is a seldom taken into account. The emphasis is on blaming and punishing them rather than figuring out the cause and stopping it happening again.

The difference is the intent. The criminal wants to do bad things while the pilot does not want anyone to get harmed.

Re: We deleted the production database by accident

#445

Post by cdaven » Tue, Oct 20, 2020, 5:51 AM UTC

jonplackett wrote: post #425

janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

It’s interesting that this is the exact opposite of how we think about crime and punishment. All criminals are like the pilot, just the person who did the action. But the reasons for them becoming criminals is a seldom taken into account. The emphasis is on blaming and punishing them rather than figuring out the cause and stopping it happening again.

The criminal has to take the punishment for his actions (and extenuating circumstances are taken into consideration), but at the same time, people, companies and society, have learned that we need protection and prevention.

So you could argue that there have been a lot of post-mortems through the ages, with great ideas thrown around on how to avoid crimes being committed (at least against me/us). It's not just about locking people up.

Re: We deleted the production database by accident

#446

Post by cdaven » Tue, Oct 20, 2020, 5:55 AM UTC

fireant wrote: post #444

jonplackett wrote: post #425

Earlier quoted context omitted.

It’s interesting that this is the exact opposite of how we think about crime and punishment. All criminals are like the pilot, just the person who did the action. But the reasons for them becoming criminals is a seldom taken into account. The emphasis is on blaming and punishing them rather than figuring out the cause and stopping it happening again.

The difference is the intent. The criminal wants to do bad things while the pilot does not want anyone to get harmed.

Sure, intent is relevant, but the example was "a suicidal pilot flying the plane into a mountain on purpose". Isn't killing all those passengers a crime?

Re: We deleted the production database by accident

#447

Post by ckuehne » Tue, Oct 20, 2020, 9:03 AM UTC

cdaven wrote: post #446

fireant wrote: post #444

Earlier quoted context omitted.

The difference is the intent. The criminal wants to do bad things while the pilot does not want anyone to get harmed.

Sure, intent is relevant, but the example was "a suicidal pilot flying the plane into a mountain on purpose". Isn't killing all those passengers a crime?

Of course it is. That is why, had the pilot survived, he/she would have faced criminal charges.

Re: We deleted the production database by accident

#448

Post by jonplackett » Wed, Oct 21, 2020, 12:56 AM UTC

Talinx wrote: post #435

jonplackett wrote: post #425

Earlier quoted context omitted.

It’s interesting that this is the exact opposite of how we think about crime and punishment. All criminals are like the pilot, just the person who did the action. But the reasons for them becoming criminals is a seldom taken into account. The emphasis is on blaming and punishing them rather than figuring out the cause and stopping it happening again.

To stop the cause from happening is not always feasible. It might also be against human rights.

Not always but a lot of crime seems like it could be avoided if more effort were put into prevention - better funding for education etc. It’s not rocket science.

Re: We deleted the production database by accident

#449

Post by jonplackett » Wed, Oct 21, 2020, 12:58 AM UTC

ckuehne wrote: post #447

cdaven wrote: post #446

Earlier quoted context omitted.

Sure, intent is relevant, but the example was "a suicidal pilot flying the plane into a mountain on purpose". Isn't killing all those passengers a crime?

Of course it is. That is why, had the pilot survived, he/she would have faced criminal charges.

It’s not one or the other, especially if there’s intent. Yes punish, but don’t scapegoat the whole systemic problems on the individual and then think you solved the problem by creating a ‘deterrence’ to others. Life doesn’t work that way. Imagine if every serious crime prompted a review and action to stop it ever happening again - imagine how much further along we’d be to a more just society.

Re: We deleted the production database by accident

#450

Post by stennie » Wed, Oct 21, 2020, 5:41 AM UTC

fma wrote: post #117

xtracto wrote: post #40

This happened to me (someone in my team) a while ago but with mongo. The production database was ssh-tunneled to the default port of the guys computer and he ran tests that cleaned the database first. Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty. What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i…

Same happened to me many years ago. QA dropped the prod db. It's been many years but if I recall, I believe in the dropdown menu of the MongoDB browser, exit & drop database were next to each other...Spent a whole night replaying the oplog. No one owned up to it, but had a pretty good idea who it was.

Which MongoDB browser or admin tool are you referring to?

I haven't seen this design in practice using MongoDB Atlas or Compass, but would hope for an "Are you really sure?" confirmation in an admin UI.