We deleted the production database by accident

Post by vanviegen » Sun, Oct 18, 2020, 7:38 AM UTC

fma wrote: post #117

xtracto wrote: post #40

This happened to me (someone in my team) a while ago but with mongo. The production database was ssh-tunneled to the default port of the guys computer and he ran tests that cleaned the database first. Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty. What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i…

Same happened to me many years ago. QA dropped the prod db. It's been many years but if I recall, I believe in the dropdown menu of the MongoDB browser, exit & drop database were next to each other...Spent a whole night replaying the oplog. No one owned up to it, but had a pretty good idea who it was.

Post by thunderrabbit » Sun, Oct 18, 2020, 7:45 AM UTC

ikiris wrote: post #193

bsder wrote: post #174

Earlier quoted context omitted.

> You can avoid this problem. No, you can't. No matter how good you are, you can always "rm -rf" your world. Yes, we can make it harder, but, at the end of the day, some human, somewhere, has to pull the switch on the stuff that pushes to prod. You can clobber prod manually, or you accidentally write an erroneous script that clobbers prod. Either way--prod is toast. The word of the day is "backups".

excuse me, but no. this is harmful bullshit. Yes, backups are vitally important, but no it is not possible to accidentally rm -rf with proper design. It's possible to have the most dangerous credentials possible and still make it difficult to do catastrophic global changes. Hell it's my job to make sure this is the case.

Post by boltefnovor » Sun, Oct 18, 2020, 7:45 AM UTC

known wrote: post #170

Sack whoever is responsible;

Post by gassius » Sun, Oct 18, 2020, 7:46 AM UTC

centimeter wrote: post #182

henearkr wrote: post #147

Earlier quoted context omitted.

When I was not yet a teetotaler, each time I was hitting my maths textbooks after a few drinks, I could not believe my level of focus, and everything was clear and obvious. Textbook pages were flying at a speed never seen. Of course the next day, when re-reading the same pages, I was always discovering that the previous day I had everything wrong, nothing was obvious, and all my reasoning when with alcohol was false…

Similar effect with psilocybin or LSD - you think you had a really profound and insightful experience, but once you think back on it you realize that (most of the time) you just got the impression that it was profound and insightful.

Post by fuzxi » Sun, Oct 18, 2020, 7:48 AM UTC

ikiris wrote: post #193

bsder wrote: post #174

Earlier quoted context omitted.

> You can avoid this problem. No, you can't. No matter how good you are, you can always "rm -rf" your world. Yes, we can make it harder, but, at the end of the day, some human, somewhere, has to pull the switch on the stuff that pushes to prod. You can clobber prod manually, or you accidentally write an erroneous script that clobbers prod. Either way--prod is toast. The word of the day is "backups".

excuse me, but no. this is harmful bullshit. Yes, backups are vitally important, but no it is not possible to accidentally rm -rf with proper design. It's possible to have the most dangerous credentials possible and still make it difficult to do catastrophic global changes. Hell it's my job to make sure this is the case.