Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

121–130 of 152 posts

Re: 'Check Your Backups Work' Day

#121

Post by hamandcheese »

elvinyung wrote: post #86
jivid wrote: post #83

Earlier quoted context omitted.

Author here. We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions. We even use MySQL as a queue for Facebook Messenger. More details about this: https://www.youtube.com/watch?v=eADBCKKf8PA https://code.facebook.com/posts/820258981365363/building-mob...

Thanks! Standardizing on MySQL because of internal expertise is definitely a great reason.

GitHub also migrated persistent data out of Redis and in to MySQL, with their expertise in MySQL as one of the motivating factors.

https://githubengineering.com/moving-persistent-data-out-of-...

Re: 'Check Your Backups Work' Day

#122

Post by scurvy »

riffraff wrote: post #20
scurvy wrote: post #16

Earlier quoted context omitted.

It's in there now. Row counts per table and checksums are written out with the backup for every table. If things don't match up, alarm in a loud and noisy way.

The point is that you could have a bug in the verification process that returns "all good" when it really isn't.

My point is that there are sanity checks built into the process. It's not difficult. There doesn't need to be a human verifying things there. Is the row count increasing over the previous backup? Check. Is the checksum different? Check. Is it non-zero? Check. And on and on.

Stop making excuses and automate testing your backups.

Re: 'Check Your Backups Work' Day

#124

Post by overcast »

misternugget wrote: post #65
timcederman wrote: post #35

When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.

You might want to check your Time Machine backups by hand from time to time using the tmutil[1] tool. My Time Machine backups in the past have been missing gigabytes of data, without me telling anything about it. And not just volatile data, like temp files or caches, but photos, music and documents. [1]: http://osxdaily.com/2012/01/21/compare-time-machine-backups-...

I use Time Machine for quick recovery stuff, and then Arq Backup to send to remote storage.

Re: 'Check Your Backups Work' Day

#125

Post by benmorris »

I discovered my Dev server DB backups weren't working after reading this. I used to consider my dev server just a place where I tossed test code to mimic production, but forgot that my DB server running on it was the only copy of potentially months of work! I have it dump nightly to my desktop now which also gets backed up locally and to the cloud.

Re: 'Check Your Backups Work' Day

#126

Post by bluGill »

hueving wrote: post #41
cyberferret wrote: post #31

Earlier quoted context omitted.

You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield. The EN…

>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially…

That is the wrong question. The real question is what did they do to ensure that this mistake cannot be made again by anybody. In the form this variation is in, the real question is what changes were made to the design so that the system cannot be over pressurized in that way by mistake.

Re: 'Check Your Backups Work' Day

#127

Post by riffraff »

scurvy wrote: post #122
riffraff wrote: post #20

Earlier quoted context omitted.

The point is that you could have a bug in the verification process that returns "all good" when it really isn't.

My point is that there are sanity checks built into the process. It's not difficult. There doesn't need to be a human verifying things there. Is the row count increasing over the previous backup? Check. Is the checksum different? Check. Is it non-zero? Check. And on and on. Stop making excuses and automate testing your backups.

so, when your check process does not report because it failed to run, how do you know? Do you have a monitor for your monitor? Does that monitor have a monitor?

There is no reason not to automate your backup tests, but there is no reason not to eyeball the check is actually working from time to time.

Re: 'Check Your Backups Work' Day

#128

Post by bluGill »

borplk wrote: post #77

Not referring to Gitlab's incident but the general problem with this stuff is too often people treat something like "make a backup" too literally. As in, they think the act of generating a backup file is the last stage of the process and they are done with it. Maybe they go the extra mile and throw it in a cronfile too. What you have to do is to consider any and all backups non-existent until you have a complete back…

There are levels. Obviously the best test is 3 backups on 3 different continents (in the future one of the backups will be on Mars), with regular tests that each site can restore from scratch.

Even the minimum level: an untested backup to the same disk is better than what most people have. That untested backup is there and protects against accidental file deletion - which will eventually become a test for most people. If the disk fails it is also something to work with. If you tell a disaster recovery service that there is a backup they can use that redundancy to their advantage: odds are physical damage isn't to both the backup and the read data and they only have to recover one. Even if the damage is to both there is something to work with.

As we move up the ladder there is more and more. An untested backup has data - give me the team a few years and we can recover it. We might have to recreate the restore from scratch, but there is something.

Remember though, the farther down the ladder you are the more expensive recovery can be. If you have tested backups on 3 continents recovery only costs a couple hours downtime - an actuary can put an exact dollar cost on this. If the price is too high you can invest in redundancy on the live system. If you have an untested backup it might be years before your team can recover it: millions of dollars in labor to recover the data and sever years of no/reduced business while they recover it. (hint: the company will go out of business because they cannot afford to recover the data)

Re: 'Check Your Backups Work' Day

#129

Post by problems »

dspillett wrote: post #45
soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

> even then it could still just stop working one day This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or tra…

> This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or transferring the backup failed.

That's awesome. I've always tested my backups by having shitty, cheap servers so bad that restoring from backup happens about once every few months.

Going to an automated restoration test is a great idea, though also a lot of work for most people.

Just checking, manually or automatically that your backups are occurring and are of a reasonable size is probably sufficient for most operations and would have caught most if not every case in the GitLab instance.

Post reply on HN