Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

71–80 of 456 posts

Re: We deleted the production database by accident

#71

Post by Izmaki »

astura wrote: post #65
jlgaddis wrote: post #58

Earlier quoted context omitted.

Take the "drop database" bit (on the production database) away from your developers, too. As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete"). If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once…

As a developer at all the jobs I've had, I've never even had access to the production database, full stop - only the server admins did. If I needed something from prod I'd go through them. I don't even consider it inconvenient.

The distance between the developer and the production database is directly proportional with the size of the company. The same is true for the amount of paperwork needed for even the simplest things. For some company sizes a feedback cycle measured in days is not good enough and may kill the business.

There is a whole lot of things that could be applied which would have prevented this, before needing to hire a separate server admin as an interface to the production database. :D

Re: We deleted the production database by accident

#73

Post by StavrosK »

muststopmyths wrote: post #7

>Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. We’re too tired to figure it out right now. The gremlins won this time. Obviously, somehow the script ran on the database host. some practices I've followed in the past to keep this kind of thing from happening: * A script that deletes all the data can never be deployed to production. * scrip…

Someone SSHed to production and forwarded the database port to the local machine to run a report, then forgot about the connection and ran the deletion script locally.

Re: We deleted the production database by accident

#74

Post by exabrial »

If you keep configuration in the environment (/etc/default/app-name) rather than in the application package, it's nearly impossible to make this mistake (especially with proper firewall rules). You can even package your config as a deb and keep it encrypted version control.

Re: We deleted the production database by accident

#75

Post by zmmmmm »

junglejoose wrote: post #32

You aren’t a real engineer until you do this. So congrats on the promotion! :)

Indeed - after incidents like this I usually say, "This is called experience that you can't pay to get for any price. Learn from it well, and value your lesson."

Re: We deleted the production database by accident

#76

Post by latch »

fideloper wrote: post #69

RDS is so very worth paying for this type of issue (in many cases, obviously $60 to multiple thousands a month isn’t great for everything). Otherwise having a binlog based backup (or WAL, I guess, but i don’t know PG that well) is critical. The key point there is they provide point in time recovery possibilities (and even the ability to rewrite history).

Barman (1) is really easy to setup and lets you avoid the many pitfalls of RDS (lower performance, high cost, no superadmin, no arbitrary extensions, waiting for releases, bad log interface).

(1) https://www.pgbarman.org/

Re: We deleted the production database by accident

#77

Post by danellis »

> after a couple of glasses of red wine, we deleted the production database by accident

> It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober.

It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might think this was emergency work, but it says "doing some late evening coding". I think this really highlights the need to separate work time from leisure time.

Re: We deleted the production database by accident

#78

Post by mbroshi »

I love this post. This sort of thing happens to everyone, most people just are not willing to be so open about it.

I was once sshed to the production server, and was cleaning up some old files that got created by an errant script, one which file was '~'. So, to clean it up, I type `rm -rf ~`.

Re: We deleted the production database by accident

#79

Post by ISL »

zmmmmm wrote: post #75
junglejoose wrote: post #32

You aren’t a real engineer until you do this. So congrats on the promotion! :)

Indeed - after incidents like this I usually say, "This is called experience that you can't pay to get for any price. Learn from it well, and value your lesson."

I'm a physicist, interested in consulting on both data analysis and precision metrology/hardware projects in general.

That said, I will happily accept consulting fees in return for deleting someone's database in prod, should they so desire.

Edit: Heck, being a white-hat licensed-to-create-mayhem chaos monkey for a few hours a week sounds pretty fun. Email in profile.

Post reply on HN