Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

131–140 of 152 posts

Re: 'Check Your Backups Work' Day

#131

Post by korzun »

michaelt wrote: post #102
toyg wrote: post #74

Earlier quoted context omitted.

it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.

I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court? What are the risks here? I'd be interested to learn more.

> I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime.

Doesn't mean it never happens.

Re: 'Check Your Backups Work' Day

#132

Post by erikbye »

ocdtrekkie wrote: post #120
erikbye wrote: post #117

Earlier quoted context omitted.

Why is it you (GitLab) and every other company has to learn from your own mistakes again and again instead of learning from others? Just a cursory glance at a few log files would have told you that your backups are not happening. Nevermind verifying actual backups... Do you have DBAs? Are they completely inept at their job? Don't answer.

Hindsight is always 20/20. Berating people who now KNOW mistakes were made accomplishes nothing.

Backup is elementary. It should not need to be learned nor manifest itself as a hindsight revelation. Frankly GitLab seems more and more like a mom and pop shop that learned Rails over a weekend. They did not even know if they had functioning backups before shit hit the fan.

Re: 'Check Your Backups Work' Day

#133

Post by ktta »

paulddraper wrote: post #9
ktta wrote: post #3

I think having a day for a whole year is a bit sparse. Some startups start and shutdown within a year. Apart from checking backups after any big code change related to backups, I think backups should be checked quarterly. It takes no more than couple hours most of the time, and as our wise said, "An ounce of prevention is worth a pound of cure"

I think that once a quarter is better, but if your start up shuts down at the end of the year, you probably don't need to worry about it

I don't think any startup knows that they're going to shutdown within the year. No one would take the time if they knew they were going to shut down soon.

Re: 'Check Your Backups Work' Day

#135

Post by dspillett »

marcosdumay wrote: post #110
dspillett wrote: post #45

Earlier quoted context omitted.

> even then it could still just stop working one day This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or tra…

> This is where automated testing helps. Now that I'm developing email stuff I get all the time the existential crisis of "how to I make sure my testing and alerting infrastructure is working?" Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you? I think I'll add periodic positive messages for my restoration procedures. Stuff like (if success and day % 23…

> Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you?

I get a daily message, so if I know if things aren't getting sent or getting through. I've been wary of setting up anything that only sends problem messages because regularly getting the "all OK" messages proves that at least that part of the infrastructure is working.

Re: 'Check Your Backups Work' Day

#136

Post by dspillett »

cuu508 wrote: post #81
dspillett wrote: post #45

Earlier quoted context omitted.

> even then it could still just stop working one day This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or tra…

> picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. This is sensible, but one problem with this is developing "blindness" for messages you receive every day. For some recurring tasks you can automate a bit further. Instead of receiving a success message every day, receive a failure message the first day ther…

> This is sensible, but one problem with this is developing "blindness" for messages you receive every day.

This can be an issue, I do get lazy about checking them when I'm busy.

The next step will be to automate checking them a bit more, rather than stopping the positive messages. A simple script that for the mail example logs in via POP/IMAP, checks for the last notification and checks the relevant timestamps. Similar for other services (is the last commit in the repo from more than 24 hours ago?). Then I get a simple "all OK" or not. I still have to run the script, perhaps hooking it to CGI and setting it to my browser home page to further remove my laziness from the equation, but I don't want to just be told when something is wrong as I'll never really trust that something going wrong won't block warning messages - I want to make checking as easy as possible though.

I may play with your service and add it as a secondary set of checks though!

Re: 'Check Your Backups Work' Day

#137

Post by vegabook »

erikbye wrote: post #117
sytse wrote: post #82

Earlier quoted context omitted.

Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.

Why is it you (GitLab) and every other company has to learn from your own mistakes again and again instead of learning from others? Just a cursory glance at a few log files would have told you that your backups are not happening. Nevermind verifying actual backups... Do you have DBAs? Are they completely inept at their job? Don't answer.

Yours is an entirely appropriate response. While I appreciate the gushing solidarity in the rest of this thread, non working backups really is an elementary IT error, accurately attributable to incompetence.

Re: 'Check Your Backups Work' Day

#138

Post by evilDagmar »

marcosdumay wrote: post #109
soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

> We should probably treat this issue as something more like a disease like high blood pressure, namely, you don't know you have it, but it is probably doing irreparable damage to your internal organs. That's why we need awareness days.

Precisely this.

Things that aren't tested can pretty much be counted on to fail a non-trivial percentage of the time. If it's going to be business critical, then it needs to be tested. (Lesson hopefully learned here)

By the same token, things which aren't monitored can also be counted upon to fail a non-trivial percentage of the time and even worse, go unnoticed. (Another teachable point here)

...but people being what they are tend to learn these things the hard way unless we've got concrete, real-world incidents like this for them to learn from.

Re: 'Check Your Backups Work' Day

#139

Post by marcosdumay »

dspillett wrote: post #135
marcosdumay wrote: post #110

Earlier quoted context omitted.

> This is where automated testing helps. Now that I'm developing email stuff I get all the time the existential crisis of "how to I make sure my testing and alerting infrastructure is working?" Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you? I think I'll add periodic positive messages for my restoration procedures. Stuff like (if success and day % 23…

> Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you? I get a daily message, so if I know if things aren't getting sent or getting through. I've been wary of setting up anything that only sends problem messages because regularly getting the "all OK" messages proves that at least that part of the infrastructure is working.

The problem with that is when you start getting too many "all OK" messages.

Getting a message must be mostly a surprise. One daily message is OK for me, but if so it may be good to tell about more than a backup. A daily digest of "everything is up and running" may indeed be something very nice to have.

And how do you ensure a team will always have someone surprised not to get a daily message? Maybe sending for may people, or requiring some acknowledge, and creating a warning over the warning channel if there isn't one. (And you certainly aren't going to send those positive messages over the warning channel, because it requires a completely different kind of action.)

Re: 'Check Your Backups Work' Day

#140

Post by elastic_church »

soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

oh I thought GitLab was testing because of this day.
Post reply on HN