Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

91–100 of 152 posts

Re: 'Check Your Backups Work' Day

#91

Post by throwaway8124 »

sytse wrote: post #67
edelans wrote: post #62

Earlier quoted context omitted.

Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

Good morning (posted from throwaway for reasons Ill describe).

I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad.

We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this, is I found out after the fact. Due to the secrecy, not even the teams knew why things failed the way they did. I had to piece it together from disjoint hearsay, and now I believe I have a competent picture.

So yes, when I read your post mortem and RCA, it reminded me greatly of what happened here as well. But we all can learn from your example. As for me, I'm posting it as a throwaway due to likely threats on my job.

Re: 'Check Your Backups Work' Day

#92

Post by abricot »

nadaviv wrote: post #70
virtualized wrote: post #58

> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_? I know th…

I would start by giving them more descriptive names... calling it db1 and db2 is a sure way to trip someone some day.

This is already on their todo in the same document.

Re: 'Check Your Backups Work' Day

#93

Post by sytse »

throwaway8124 wrote: post #91
sytse wrote: post #67

Earlier quoted context omitted.

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

Good morning (posted from throwaway for reasons Ill describe). I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad. We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this,…

I agree that 6 hours is way too much.

Re: 'Check Your Backups Work' Day

#94

Post by tawayway »

ocdtrekkie wrote: post #56
tawayway wrote: post #32

Earlier quoted context omitted.

> Imagine if you're the CEO of Gitlab and seeing this right now I kind of hope the CEO of Gitlab isn't reading HN right now

If he's getting some much needed sleep right now, he'll probably read it tomorrow. The CEO of GitLab is really active on HN, and a pretty classy guy.

I didn't mean as a slight against him, just kidding :)

Re: 'Check Your Backups Work' Day

#97

Post by wila »

timcederman wrote: post #35

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

Time Machine backups are great for a lot of use cases, but beware to not trust your virtual machines to Time Machine [0].

You might end up disappointed if you do.

[0] https://kb.vmware.com/kb/1013628

Re: 'Check Your Backups Work' Day

#98

Post by acdha »

Joe8Bit wrote: post #78
kator wrote: post #75

In my day we used to develop "Disaster Recovery" programs. They were massive, and we tested on a regular basis including renting massive systems from IBM and flying the team to the IBM data center to run a full restore of everything. End business users had to login afterwards and sign off. I understand we "live in a different world" is the favorite motto these days. But do we really? If anything data is bigger, more…

I don't recognise the 'different world' you describe. Practiced disciplined DR is a key part of modern software engineering at large and small companies across the startup/enterprise spectrum (in my experience). The popularity of tools like Netflix's Chaos Monkey/Gorilla/Kong server as testament to that. That's not to say all companies do it (it seems GitLab didn't) but the tone of your comment doesn't reflect a lot…

I think the key is knowing that this is routine at a subset of shops while most others are winging it to various degrees — and always has been.

If you worked at a responsible place decades ago you might be aghast at what happens at a random sampling of companies now, but the same would have been true at a random sampling decades ago. The difference is that unless you were a customer or the outage was especially prominent you probably never would have heard about it.

Post reply on HN