Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our Server’s Hard Drive is Dead. We didn’t have a backup.

blog.method.ac

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#81

Post by rorrr2 »

nwilkens wrote: post #76
rorrr2 wrote: post #68

Earlier quoted context omitted.

Good luck with your below-the-minimum-wage admin.

I can understand with those numbers why you may think that. These numbers are generally per server and fit a small shop pretty well. We also work with many companies that have hundreds of servers, and the rate is multiplied by each server. I could have been clearer in my initial post. Additionally, for clarification about your wage comment -- we pay above average US sysadmin wages, and just posted a job for our 7th f…

> These numbers are generally per server

Well, that's a quite important detail you had omitted. If you start dividing salaries by the number of servers, all kinds of crazy numbers come up.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#82

Post by thaumaturgy »

a. My business has a partnership with a good data recovery outfit. We might be able to get you a good deal on a data recovery if you want to try going that route.

b. It takes a particular kind of personality to be good at sysadmin work. (And a lot of trial-and-error -- I just recently had to do an emergency server build due to a Debian update whoops, and I've been doing this stuff for a while.)

c. I usually recommend BackupPC (http://backuppc.sourceforge.net/) for easy set-it-and-forget-it backup infrastructure. It's compatible with everything, it will notify you if there are problems, it does pooling and de-duplication and compression, it's fast and reliable, and you can usually store months of backups on a small offsite server. I store 12 months of all hosted and customer data with it, and we've used it to meet other clients' needs too.

d. If you need affordable help, let me know. I'm way too cheap, and I do this stuff all day, every day. I opened a business specifically to address problems like this: needs something, money is a problem.

That goes for anybody else too. If your lack of backups is keeping you awake at night, or if you've suddenly outgrown your infrastructure, or if looking at config files gives you an ulcer, get in touch with me. I'll help you out.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#83

Post by foobarbazqux »

femto wrote: post #78
foobarbazqux wrote: post #70

Earlier quoted context omitted.

Yeah, that's true. But then do you really think those guys are spamming HN at 8 p.m.? I wish I had said, if you ignore the advertising, it's a great resource. If you apply it to proposed backup solutions, it's an effective means to find out if they are viable.

I said it for you, since a reader would probably be interested in that piece of information. I apologise for the cynical comment about the account being 3 minutes old, as that is ad hominem and being new to HN shouldn't come into it. Welcome to HN.

Oh it's okay. Thanks for pointing it out. Truth is I lost my password and I hadn't given them my email, so I just made a new account. (I'm not upset about this.)

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#84

Post by cdvonstinkpot »

protomyth wrote: post #62
gregd wrote: post #60

Earlier quoted context omitted.

Actually it's been my experience that SSDs are less reliable than HDDs...

And for the love of all you hold holy, do not ever put drives bought in the same batch in the RAID at the same time. They tend to fail at the same time. Check those serial numbers first.

That's most unfortunate to hear, because I just bought 10x used Intel X25-E SSDs for my server because SSDLife.exe said they only had 6 months of use, with 99% of their life left & an expected runtime of 10 more years.

With these drives typically around $750/each, it'll be difficult if not impossible to find deals on 8 separate lots of _good_ used drives.

I can only afford them when I see an exceptional deal on eBay, and feel lucky to have found these I just got for $200/each. I strongly doubt I'll be able to afford to implement 1 drive per lot with my server like your warning implies would be a good thing to do.

It seems you'd have to buy them all new at different times of the year to be able to implement such a thing, which I certainly can't afford.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#85

Post by protomyth »

cdvonstinkpot wrote: post #84
protomyth wrote: post #62

Earlier quoted context omitted.

And for the love of all you hold holy, do not ever put drives bought in the same batch in the RAID at the same time. They tend to fail at the same time. Check those serial numbers first.

That's most unfortunate to hear, because I just bought 10x used Intel X25-E SSDs for my server because SSDLife.exe said they only had 6 months of use, with 99% of their life left & an expected runtime of 10 more years. With these drives typically around $750/each, it'll be difficult if not impossible to find deals on 8 separate lots of _good_ used drives. I can only afford them when I see an exceptional deal on eBay,…

Well, I went with the buy one drive a month from each of my vendors for my RAID.

Use them, but take some precautions. Do real backups and test the backup to make sure it will actually restore. Replace blown drives very quickly. I had a group of C4 SSDs that tanked within 24 hours of each other. I was not amused and thus learned what I need to do in the future.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#86

Post by protomyth »

gregd wrote: post #64
protomyth wrote: post #62

Earlier quoted context omitted.

And for the love of all you hold holy, do not ever put drives bought in the same batch in the RAID at the same time. They tend to fail at the same time. Check those serial numbers first.

If you really want to have some fun, test the hot swap capabilities in the middle of a very busy, data centric day.

We do that before the equipment goes into production. We do not tempt Mr. Murphy to visit us in production.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#87

Post by toomuchtodo »

nwilkens wrote: post #77
toomuchtodo wrote: post #75

Earlier quoted context omitted.

Expert admins are in the $100-150/hr range when they're consulting, and $100K+ salary range when hired. $100-200/month? Oh lord.

Agreed with the consulting price point, and we charge this. But we also let you get in the door for a few hours a month at a reasonable rate. It's been working for many years now.

You're also based on Michigan a little south of Detroit, so I'm sure you can get away with a lower base rate based on the cost of living there.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#88

Post by nwilkens »

toomuchtodo wrote: post #87
nwilkens wrote: post #77

Earlier quoted context omitted.

Agreed with the consulting price point, and we charge this. But we also let you get in the door for a few hours a month at a reasonable rate. It's been working for many years now.

You're also based on Michigan a little south of Detroit, so I'm sure you can get away with a lower base rate based on the cost of living there.

Sorta, except we generally hire where the talent is available -- Anywhere USA.
Post reply on HN