Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

231–240 of 456 posts

Re: We deleted the production database by accident

#231

Post by dingdingdang »

toredash wrote: post #201
unnouinceput wrote: post #199

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

2 cents his hosts file points localhost to the prod db IP

Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas").

Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") that way local specific scripts can be stripped even if in same repo.

Re: We deleted the production database by accident

#232

Post by Biganon »

defen wrote: post #15
xnyan wrote: post #11

localhost is an abstraction, it's a non-routable-outside-your-machine network...except it's not. It's nothing more than normal TCP traffic except with a message to the OS and other programs that whatever is on that local computer network, you don't want it routed outside the local computer. There's absolutely nothing stopping anything with access to localhost from routing it anywhere that process wants. Does not even…

The fact that the production db has the same username/password as the development one is perhaps more troubling.

They say in the article that it's not the case.

Re: We deleted the production database by accident

#233

Post by radu_floricica »

toredash wrote: post #201
unnouinceput wrote: post #199

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

2 cents his hosts file points localhost to the prod db IP

I'm betting on a tunnel, myself. And grandparent is probably wrong, they most likely have dedicated mysql machines so "localhost" will never be the db.

Re: We deleted the production database by accident

#235

Post by skytreader »

unnouinceput wrote: post #199

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

> Instead one of your gremlins ran this function directly on the production machine.

Exactly my first hypothesis too. But then keepthescore claims,

> of course we use different passwords and users for development and production.

How would this hypothesis explain that?

---

Metadialogue:

John Watson: "I deduce that someone changed the dev config source so that it uses the production config values."

Sherlock Holmes: "My dear Watson, while that is sensible, it seems to me that the balance of probability leans towards their production instance also having the development access credentials."

---

Just my way of saying, I think this case isn't as shut and closed as most comments (including parent) imply. I personally find the /etc/host mapping a likelier hypothesis but even that can't explain how different credentials failed to prevent this. Without more details coming from a proper investigation, we are just piling assumptions on top of assumptions. We are making bricks, without enough clay, as Holmes would say.

Re: We deleted the production database by accident

#236

Post by fauigerzigerk »

john_moscow wrote: post #139
candiddevmike wrote: post #129

Earlier quoted context omitted.

Anyone reading the above: please don't do this. Git is not made for database backups, use a real backup solution like WAL archiving or dump it into restic/borg. Your git repo will balloon at an astronomical rate, and I can't imagine why anyone would diff database backups like this.

It really depends on your database size. This works just fine for ~300MB databases. Git gc takes pretty good care of the fluff and once every couple of years I reset the repository to prune the old snapshots. The big plus is that you can reuse your existing git infrastructure, so the marginal setup costs are minimal. You can always switch to a more specialized solution if the repository size starts bugging you, but d…

>It really depends on your database size.

This isn't just about size though. You're storing all customer data on all developer machines. You're just one stolen laptop away from your very own "we take the security of our customers' data very seriously" moment.

Re: We deleted the production database by accident

#237

Post by heavenlyblue »

aszen wrote: post #184

I had a narrow escape once doing something fancy with migrations. We had several MySQL string columns as long text type in our database but they should have been varchar(255) or so. So I was assigned to convert these columns to their appropriate size. Being the good developer I was, I decided to download a snapshot of the prod database locally and checked the maximum string length we had for each column via a script.…

And you didn’t even bother to do a query of the actual maximum length value of the columns you were mutating? Or at least query and see the text in there?

Basically you just blindly ran the migration on the data and checked if it didn’t fail?

The lesson here is not about cleverness unfortunately.

Re: We deleted the production database by accident

#238

Post by detaro »

heavenlyblue wrote: post #237
aszen wrote: post #184

I had a narrow escape once doing something fancy with migrations. We had several MySQL string columns as long text type in our database but they should have been varchar(255) or so. So I was assigned to convert these columns to their appropriate size. Being the good developer I was, I decided to download a snapshot of the prod database locally and checked the maximum string length we had for each column via a script.…

And you didn’t even bother to do a query of the actual maximum length value of the columns you were mutating? Or at least query and see the text in there? Basically you just blindly ran the migration on the data and checked if it didn’t fail? The lesson here is not about cleverness unfortunately.

The comment clearly states that they did.

Re: We deleted the production database by accident

#239

Post by jnsaff2 »

bstar77 wrote: post #134
danellis wrote: post #77

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

I am not a drinker myself (drink 1-3 times a year), but in the past I have coded while slightly buzzed on a few occasions. I could not believe the level of focus I had. I never investigated it further, but I'm pretty sure the effects of alcohol on our coding abilities is not nearly as bad as it affects our motor skills. Imo, fatigue is far worse.

https://xkcd.com/323/

Re: We deleted the production database by accident

#240

Post by skrebbel »

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault".

This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously sorry, we're doing everything in our power yadayada" isn't going to help anybody.

Yes, there's all kinds of things they could have done to prevent this from happening. Yes, some of the things they did (not) do were clearly mistakes that a seasoned DBA or sysadmin would not make. Possibly they aren't seasoned DBAs or sysadmins. Or they are but they still made a mistake.

This stuff happens. It sucks, but it still does. Get over yourselves and wish these people some luck.

Post reply on HN