Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

111–120 of 152 posts

Re: 'Check Your Backups Work' Day

#111

Post by marcosdumay »

virtualized wrote: post #58

> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_? I know th…

> Why does he have SSH access there?

Because when something goes wrong he has to be able to discover what it was.

> Why do they test their database replication in production?

Many places do that. With free databases there is simply no reason to, but the practices are inherited from the non-free databases best practices arsenal.

People, if you are using postgres and have a cluster in production, make sure you also have the script that creates this cluster in a VCS and is able to create an equal cluster on virtual machines in your computer.

> Why are they fire-fighting by changing random database settings _in production_?

Oh, that's because they are fire-fighting. The problem is that when you get a problem you know nothing about, you don't really know what to replicate in another environment to reliably test things there. The most you can do is verify that your changes aren't harmful, the real test is always in production.

Re: 'Check Your Backups Work' Day

#112

Post by harshreality »

sytse wrote: post #82
ocdtrekkie wrote: post #56

Earlier quoted context omitted.

If he's getting some much needed sleep right now, he'll probably read it tomorrow. The CEO of GitLab is really active on HN, and a pretty classy guy.

Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.

The lesson here has been learned over and over again, painfully.

For every backup system you have, test restores, from scratch, periodically. The more critical the data is to your business, the more frequently and more automated you want those backup checks.

Of course you also want procedures to try to prevent tired admins from deleting production databases. To the greatest degree possible, implement systems to prevent manual tampering with production data; often there's an alternative. But databases can get broken or corrupt without admin interference, so preventing manual database removal or corruption is not an ultimate solution.

Re: 'Check Your Backups Work' Day

#114

Post by arca_vorago »

I would like to take this moment to impart to all of you concerned about your business backups to make sure you are enabling your systems administrator with the budget, tools, personnel, and backing from management he/she needs to get the job done.

I have seen far too often one man miracle teams swimming in technical debt solving problems constantly but failing to have the time to play the political games needed to push for the kinds of changes they need. Obviously small operations setups are different, but for example, I've seen a ~250 person 6 branch business have 1 senior and 1 junior part time sysadmin, while his requests for a budget and personnel were constantly denied, and so he said his backup system worked but he knew it wasn't as good as he could make it. He eventually quit in frustration. He was a great sysadmin but didn't play enough politics and therefor he failed and his management failed him, all the while jeapordizing the business. Please don't do this to your sysadmin.

CTOs and CIOs, please take a moment to ask your sysadmin what things they need they haven't been able to convince you of yet, and see if you can compromise or otherwise try to lend their arguments importance.

In all but the leanest of SV web startup land, sysadmin are the backbone that keeps your company running. Don't neglect or forget them.

If you do, one day that backup may fail, or a cryptovariant will hit the server, and although you will scapegoat your sysadmin, it will have truly been your fault.

Re: 'Check Your Backups Work' Day

#115

Post by throwaway1236t4 »

sytse wrote: post #67
edelans wrote: post #62

Earlier quoted context omitted.

Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

I also would ask Why are we running our own Postgre setup and not using RDS? And then Why do we not have production Postgre DBAs on staff that would do this rather than an engineer?

Re: 'Check Your Backups Work' Day

#116

Post by toyg »

michaelt wrote: post #102
toyg wrote: post #74

Earlier quoted context omitted.

it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.

I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court? What are the risks here? I'd be interested to learn more.

> I've never heard of anyone successfully suing AWS/Google/Azure over lost data

Those companies have huge legal departments writing iron-clad contracts and stocking a lot of very sharp knives for any such eventuality. Smaller companies make for much easier targets.

> even if your notes are kept private, it'll still come out in discovery

Those notes might have been lost by then - that is, if their existence at any given time is even known. It is perfectly reasonable for people working quickly after an outage, to not actually write down every step or observation they make.

> What are the risks here?

Not prison, but you might be forced to pay a bunch if someone manages to get a ruling against you for negligence or the likes.

Re: 'Check Your Backups Work' Day

#117

Post by erikbye »

sytse wrote: post #82
ocdtrekkie wrote: post #56

Earlier quoted context omitted.

If he's getting some much needed sleep right now, he'll probably read it tomorrow. The CEO of GitLab is really active on HN, and a pretty classy guy.

Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.

Why is it you (GitLab) and every other company has to learn from your own mistakes again and again instead of learning from others? Just a cursory glance at a few log files would have told you that your backups are not happening. Nevermind verifying actual backups...

Do you have DBAs? Are they completely inept at their job? Don't answer.

Re: 'Check Your Backups Work' Day

#119

Post by throwusawayus »

HarrisonFisk wrote: post #5

This is every day at Facebook! https://code.facebook.com/posts/1007323976059780/continuous-...

A couple of months ago, in a public chatroom for DBAs, one of GitLab's engineering leads made some very insulting comments about Facebook engineers & eng practices. Pretty ironic.

I want to feel bad for GibLab, but it's really, really hard when they hire people like that.

Re: 'Check Your Backups Work' Day

#120

Post by ocdtrekkie »

erikbye wrote: post #117
sytse wrote: post #82

Earlier quoted context omitted.

Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.

Why is it you (GitLab) and every other company has to learn from your own mistakes again and again instead of learning from others? Just a cursory glance at a few log files would have told you that your backups are not happening. Nevermind verifying actual backups... Do you have DBAs? Are they completely inept at their job? Don't answer.

Hindsight is always 20/20. Berating people who now KNOW mistakes were made accomplishes nothing.
Post reply on HN