Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

111–120 of 456 posts

Re: We deleted the production database by accident

#111

Post by bigbubba »

cle wrote: post #102
bigbubba wrote: post #95

Earlier quoted context omitted.

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

Don’t be absurd. Of course there are costs and tradeoffs to guardrails, and you have to balance the tradeoffs based on your requirements. This person had to publicly apologize to their customers. One or two low-cost guardrails could have prevented it and would probably have been worth the cost.

Is that absurd? Systems should prevent mistakes, unless the part needed to implement that is a hundred dollars or so? That seems like quite the walk-back. Courts order alcoholics to install these things, they're established available technology. What tradeoffs are you balancing here?

Re: We deleted the production database by accident

#112

Post by mgkimsal »

john_moscow wrote: post #81

Just my 2 cents. I run a small software business that involves a few moderately-sized databases. The day I moved from a fully managed hosting to a Linux VPS, I have crontabbed a script like this to run several times a day: for db in `mysql [...] | grep [...]` do mysqldump [...] > $db.sql done git commit -a -m "Automatic backup" git push [backup server #1] git push [backup server #2] git push [backup server #3] git gc…

Any issues with the privacy aspect of that data that's stored in multiple git repos? PII and such?

Re: We deleted the production database by accident

#113

Post by bufferoverflow »

john_moscow wrote: post #81

Just my 2 cents. I run a small software business that involves a few moderately-sized databases. The day I moved from a fully managed hosting to a Linux VPS, I have crontabbed a script like this to run several times a day: for db in `mysql [...] | grep [...]` do mysqldump [...] > $db.sql done git commit -a -m "Automatic backup" git push [backup server #1] git push [backup server #2] git push [backup server #3] git gc…

You should really compress them instead of dumping them raw into Git. LZ4 or ZStandard are good.

Re: We deleted the production database by accident

#114

Post by adzm »

bufferoverflow wrote: post #113
john_moscow wrote: post #81

Just my 2 cents. I run a small software business that involves a few moderately-sized databases. The day I moved from a fully managed hosting to a Linux VPS, I have crontabbed a script like this to run several times a day: for db in `mysql [...] | grep [...]` do mysqldump [...] > $db.sql done git commit -a -m "Automatic backup" git push [backup server #1] git push [backup server #2] git push [backup server #3] git gc…

You should really compress them instead of dumping them raw into Git. LZ4 or ZStandard are good.

But then you don't have good diffs.

Re: We deleted the production database by accident

#115

Post by ethbr0 »

dnautics wrote: post #107
sushshshsh wrote: post #105

Earlier quoted context omitted.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

> Please don't use csv

Could you elaborate? I'm interested in the specific reasons.

Re: We deleted the production database by accident

#116

Post by pontifier »

im3w1l wrote: post #109
yunruse wrote: post #30

Earlier quoted context omitted.

I feel that computers make it easier for this danger to be more indirect, however. The examples you give are physical, and even the youngest of child would likely recognise they are not regular items. A production database, meanwhile, is visually identical to a test database, if measures are not made to make it distinct. Adults though we may be, we're human, and humans can make really daft mistakes without the right…

So, silly idea. What if, to work on the production database, you had to go into the special production room, colored in the special production color, scented with the special production perfume, and sit on a just tiny bit uncomfortable production chair. Basically make it clear even to the caveman brain that things are different.

I actually really like this idea... but who am I kidding, it's a luxury I don't have time for when I've got to fix stuff.

Re: We deleted the production database by accident

#117

Post by fma »

xtracto wrote: post #40

This happened to me (someone in my team) a while ago but with mongo. The production database was ssh-tunneled to the default port of the guys computer and he ran tests that cleaned the database first. Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty. What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i…

Same happened to me many years ago. QA dropped the prod db. It's been many years but if I recall, I believe in the dropdown menu of the MongoDB browser, exit & drop database were next to each other...Spent a whole night replaying the oplog.

No one owned up to it, but had a pretty good idea who it was.

Re: We deleted the production database by accident

#118

Post by bArray »

Hmm, there seems to be some holes in their system. A database might go down for any reason.

I also have daily backups, but I write logs (locally and regularly copy from the production server) for all database actions to disk for the purpose of checking through them if something goes wrong, or having the option to replay them encase something like this happens. SO you have your database backups as "save points" and the logs to replay all the "actions" for that day.

Re: We deleted the production database by accident

#119

Post by emodendroket »

This post is embarrassing. "yeah we were drinking and accidentally nuked the prod DB. Not sure why. Shit happens!" Who would read this and think they should trust this company? Any number of protections could have been taken to prevent this and production access in any state other than fully alert and attentive shouldn't happen unless it is absolutely necessary for emergency reasons

Re: We deleted the production database by accident

#120

Post by judge2020 »

ethbr0 wrote: post #115
dnautics wrote: post #107

Earlier quoted context omitted.

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

> Please don't use csv Could you elaborate? I'm interested in the specific reasons.

A quick rundown https://sqlite.org/aff_short.html
Post reply on HN