Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our Server’s Hard Drive is Dead. We didn’t have a backup.

blog.method.ac

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#61

Post by harshreality »

JPKab wrote: post #15

It's noble of you to come clean and own your mistake, but let me say this over and over: You should never, ever provide an environment that stores people's hard work without having professionals who know how to safeguard it. If it makes you feel any better, I recently had to clean up a mess in a huge enterprise IT shop, (if I were to name the organization you would immediately know them) involving hundreds of thousan…

Running cronjob backups and looking at them in passing to see that they look like valid backups is not sufficient for any serious website or web service.

Automated backups need automated backup restoration and testing. Otherwise, the backups might not be created properly, or they might be perfect backups that have some hidden error that will cause them to fail when they're put to use.

As an example, Jeremiah Wilton's self-case study on Amazon's Oracle database problem in 1997. http://www.bluegecko.net/download/disaster-diary.pdf

Other than the one missed backup, backup procedures were fine. An Oracle bug caused Oracle to refuse to start due to a database format/schema change weeks earlier. TESTING backups would have caught the error, and allowed them to fix it before they took down their production database and triggered the bug on the next attempt to start it again.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#62

Post by protomyth »

gregd wrote: post #60
cdvonstinkpot wrote: post #56

I wonder if it still would've happened if they were SSDs... I tend to think they're safer due to no moving parts.

Actually it's been my experience that SSDs are less reliable than HDDs...

And for the love of all you hold holy, do not ever put drives bought in the same batch in the RAID at the same time. They tend to fail at the same time. Check those serial numbers first.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#64

Post by gregd »

protomyth wrote: post #62
gregd wrote: post #60

Earlier quoted context omitted.

Actually it's been my experience that SSDs are less reliable than HDDs...

And for the love of all you hold holy, do not ever put drives bought in the same batch in the RAID at the same time. They tend to fail at the same time. Check those serial numbers first.

If you really want to have some fun, test the hot swap capabilities in the middle of a very busy, data centric day.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#65

Post by ironchef »

ams6110 wrote: post #6
ironchef wrote: post #3

On the plus side, this will probably only ever happen to you once. Once you've felt the pain, you'll never let it happen again.

However remember that backups are only half of the disaster recovery picture. You need to have a tested restore process ready to go as well.

Automated and tested is even better :)

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#66

Post by femto »

foobarbazqux wrote: post #63

This site is an excellent way to find out if you've covered all your bases in your backup protocol: http://www.taobackup.com/

Note: the website is an extended advertisement for a piece of backup software, and the user account was created 3 minutes before the comment was posted.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#67

Post by jameskilton »

ashray wrote: post #46
GalacticDomin8r wrote: post #36

Earlier quoted context omitted.

It also isn't a backup unless it's tested.

You're right and the cosmos has just proved it. I just went in to check my backups and found some that hadn't been running since Nov. 19th 2012. Well, lesson learnt :) - gotta test those backups!

I can't recommend https://deadmanssnitch.com/ highly enough. This would have told you your backups weren't working a long time ago!

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#70

Post by foobarbazqux »

femto wrote: post #66
foobarbazqux wrote: post #63

This site is an excellent way to find out if you've covered all your bases in your backup protocol: http://www.taobackup.com/

Note: the website is an extended advertisement for a piece of backup software, and the user account was created 3 minutes before the comment was posted.

Yeah, that's true. But then do you really think those guys are spamming HN at 8 p.m.?

I wish I had said, if you ignore the advertising, it's a great resource. If you apply it to proposed backup solutions, it's an effective means to find out if they are viable.

Post reply on HN