Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

141–150 of 456 posts

Re: We deleted the production database by accident

#141

Post by john_moscow »

mgkimsal wrote: post #112
john_moscow wrote: post #81

Just my 2 cents. I run a small software business that involves a few moderately-sized databases. The day I moved from a fully managed hosting to a Linux VPS, I have crontabbed a script like this to run several times a day: for db in `mysql [...] | grep [...]` do mysqldump [...] > $db.sql done git commit -a -m "Automatic backup" git push [backup server #1] git push [backup server #2] git push [backup server #3] git gc…

Any issues with the privacy aspect of that data that's stored in multiple git repos? PII and such?

These are private repos on private machines communicating over SSL on non-standard ports with properly configured firewall. The risk is always there, but it's low.

Re: We deleted the production database by accident

#142

Post by bstar77 »

emodendroket wrote: post #119

This post is embarrassing. "yeah we were drinking and accidentally nuked the prod DB. Not sure why. Shit happens!" Who would read this and think they should trust this company? Any number of protections could have been taken to prevent this and production access in any state other than fully alert and attentive shouldn't happen unless it is absolutely necessary for emergency reasons

I think it's kind of funny they chose to post this story rather than do a typical post mortem.

Re: We deleted the production database by accident

#143

Post by candiddevmike »

john_moscow wrote: post #139
candiddevmike wrote: post #129

Earlier quoted context omitted.

Anyone reading the above: please don't do this. Git is not made for database backups, use a real backup solution like WAL archiving or dump it into restic/borg. Your git repo will balloon at an astronomical rate, and I can't imagine why anyone would diff database backups like this.

It really depends on your database size. This works just fine for ~300MB databases. Git gc takes pretty good care of the fluff and once every couple of years I reset the repository to prune the old snapshots. The big plus is that you can reuse your existing git infrastructure, so the marginal setup costs are minimal. You can always switch to a more specialized solution if the repository size starts bugging you, but d…

Git GC won't do anything here unless you're deleting commits or resetting the repo constantly. Every commit will keep piling up, and you will never prune anything like you would a traditional backup tool. The day you do decide to start pruning things, expect your computer to burst into flames as it struggles to rewrite the commit history!

Using a real database backup solution isn't a premature optimization, it's basic system administration.

Re: We deleted the production database by accident

#144

Post by bachmeier »

hannofcart wrote: post #46

It takes a great deal of integrity to admit that you deleted a database because you were mucking around in your infra after red wine. And it bodes well for your firm that that doesn't get you fired either. These things happen to the best of us but having dealt with it responsibly and honestly as a team is something you can be proud of IMO.

> And it bodes well for your firm that that doesn't get you fired either.

Maybe there's more than one interpretation of "bodes well" but not knowing how to do the one thing customers were paying you to do isn't consistent with my definition.

> having dealt with it responsibly and honestly as a team is something you can be proud of IMO.

"We were drinking wine and deleted the database and now your data's gone LOL" is not something that should make you proud.

Re: We deleted the production database by accident

#145

Post by jzer0cool »

fma wrote: post #123
jzer0cool wrote: post #86

Could someone explain more what caused the prod wipe? The snip here indicates it is using a 'dev' credential (it is a different pass than prod right?) - how does a db connection occur at all?

Good catch. Wouldn't surprise me if there's 1 username/password for all their DB environments.

When looking at the snip again, there is a note that says it uses different a username / password for prod & dev.

I can't see how the OP indicates making a connection to db without the correct credentials to begin with.

Re: We deleted the production database by accident

#146

Post by lysp »

I had a client who had prod database access due to it being hosted internally. They called up saying "their system is no longer working".

After about an hour of investigation, I find one of the primary database tables is empty - completely blank.

I then spend the next hour looking through code to see if there's any chance of a bug that would wipe their data and couldn't find anything that would do that.

I then had to make "the phone call" to the client saying that their primary data table had been wiped and I didn't know what we did wrong.

Their response: "Oh I wrote a query and accidentally did that, but thought I stopped it".

Re: We deleted the production database by accident

#147

Post by henearkr »

bstar77 wrote: post #134
danellis wrote: post #77

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

I am not a drinker myself (drink 1-3 times a year), but in the past I have coded while slightly buzzed on a few occasions. I could not believe the level of focus I had. I never investigated it further, but I'm pretty sure the effects of alcohol on our coding abilities is not nearly as bad as it affects our motor skills. Imo, fatigue is far worse.

When I was not yet a teetotaler, each time I was hitting my maths textbooks after a few drinks, I could not believe my level of focus, and everything was clear and obvious. Textbook pages were flying at a speed never seen.

Of course the next day, when re-reading the same pages, I was always discovering that the previous day I had everything wrong, nothing was obvious, and all my reasoning when with alcohol was false because simplistic and oblivious of any mathematical rigor.

Re: We deleted the production database by accident

#148

Post by luord »

This is what nightmares are made of.

> We’ve learned that having a function that deletes your database is too dangerous to have lying around.

Indeed, anything that might compromise the data, anything that might involve deletion anyway, should require manual confirmation whether you manage the database or it's a service provided.

Sadly, I learned this the hard way too, but at least it was a single column with a non-critical date and not the entire database.

Re: We deleted the production database by accident

#150

Post by TomVDB »

bigbubba wrote: post #111
cle wrote: post #102

Earlier quoted context omitted.

Don’t be absurd. Of course there are costs and tradeoffs to guardrails, and you have to balance the tradeoffs based on your requirements. This person had to publicly apologize to their customers. One or two low-cost guardrails could have prevented it and would probably have been worth the cost.

Is that absurd? Systems should prevent mistakes, unless the part needed to implement that is a hundred dollars or so? That seems like quite the walk-back. Courts order alcoholics to install these things, they're established available technology. What tradeoffs are you balancing here?

It's a free website to keep scoreboards. Not a mission critical nuclear missile launcher.

(But if your mission critical system relies on that scoreboard website, that's on you...)

Post reply on HN