Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

31–40 of 152 posts

Re: 'Check Your Backups Work' Day

#31

Post by cyberferret »

yeukhon wrote: post #28
soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

I think this is a big slap to company when they need to talk to their enterprise customers. I am not sure how we can really prevent this from happening by implementing an industry standard.

You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield.

The ENTIRE Saturn 5 rocket had to be wheeled back to the hangar and the CM dismantled and rebuilt, resulting in a month long delay in the launch.

When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again."

I am willing to bet that Gitlab is now a company that will never slacken off their backup checking in the future.

Re: 'Check Your Backups Work' Day

#32

Post by tawayway »

soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

> Imagine if you're the CEO of Gitlab and seeing this right now

I kind of hope the CEO of Gitlab isn't reading HN right now

Re: 'Check Your Backups Work' Day

#33

Post by mrmondo »

soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

I could not agree more @soheil, let me also say that Sid aka @sytse is not only a top notch CEO but he's also a hard working, caring individual who truly not only cares about the product but the whole community so, so much.

Re: 'Check Your Backups Work' Day

#34

Post by mrmondo »

ktta wrote: post #3

I think having a day for a whole year is a bit sparse. Some startups start and shutdown within a year. Apart from checking backups after any big code change related to backups, I think backups should be checked quarterly. It takes no more than couple hours most of the time, and as our wise said, "An ounce of prevention is worth a pound of cure"

Indeed, If one day a year is what it takes for you to remember to care about backups, you probably shouldn't be involved with backups. This could happen to anyone from a person losing their home photo collection, to a hospital missing critical data within a patient management system - I know because I've been involved or seen first hand both of these exact scenarios. It's what we learn from our mistakes that defines our future, not the people telling us we made them.

Re: 'Check Your Backups Work' Day

#35

Post by timcederman »

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

Re: 'Check Your Backups Work' Day

#36

Post by mosselman »

timcederman wrote: post #35

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

Having Time Machine report to you is 'checking' as well. Nobody said anything about any manual labour as far as I can tell.

Re: 'Check Your Backups Work' Day

#37

Post by cookiecaper »

timcederman wrote: post #35

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

Big difference between consumer-level backup of phones and PCs and enterprise backup for servers crammed with custom software.

Re: 'Check Your Backups Work' Day

#38

Post by timcederman »

cookiecaper wrote: post #37
timcederman wrote: post #35

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

Big difference between consumer-level backup of phones and PCs and enterprise backup for servers crammed with custom software.

Why should "check your backups day" apply to only enterprise backup? I've seen it repeated as gospel many times on Hacker News to folks who've lost backups that it was their own fault for not checking them. My point is we should focus on a software solution to making this more robust, and not blaming people.

Re: 'Check Your Backups Work' Day

#39

Post by timcederman »

mosselman wrote: post #36
timcederman wrote: post #35

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

Having Time Machine report to you is 'checking' as well. Nobody said anything about any manual labour as far as I can tell.

In the thread about the Gitlab backups, manual checking (not "labour") was suggested at least twice.

Re: 'Check Your Backups Work' Day

#40

Post by gorodetsky »

mrmondo wrote: post #33
soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

I could not agree more @soheil, let me also say that Sid aka @sytse is not only a top notch CEO but he's also a hard working, caring individual who truly not only cares about the product but the whole community so, so much.

Wholeheartedly agree.

Incidents are inevitable and it's important to have a proper RCA/Service-Disruption process in place to handle those.

As mentioned on the other thread, maybe Gitlab doesn't have enough operational/SRE expertise in-house yet, but that was the case in every fast-growing company I've worked for last decade.

Post reply on HN