Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our Server’s Hard Drive is Dead. We didn’t have a backup.

blog.method.ac

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#11

Post by ashray »

gregd wrote: post #8
jthol wrote: post #4

Earlier quoted context omitted.

What do you mean?

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

The most basic backup would be to have a remote backup of your files and databases. That's a simple shell script and cron job combination. I can understand that one may not get around to setting it up (I had a massive project and only started doing this after about 7 years of running it..) but it's definitely very important and doesn't need a ton of Sysadmin experience or setup time.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#12

Post by jefe78 »

"We messed up bad. We launched without having a backup procedure in place, and without the resources to make it happen. This was a hard-learned lesson that won’t happen again. We have no one except ourselves to blame."

You know how you messed up? By not using something like AWS - EC2 - Snapshots. Or even S3 or Glacier. What is this trend of devs doing Operations? As a Sysadmin with a Compsci/dev background, it blows my mind constantly.

Great, you know how to move around the CLI, but are you versed in how to maintain a proper and robust system?

Also, why weren't you using something like SES for your email alerts?

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#13

Post by jrussbowman »

gregd wrote: post #2

"We need to get programming talent on-board." Sounds to me like they still haven't learned their lesson...

glad I wasn't the only one who had that as a first thought. as an ops guys I wasn't sure if I should be offended or just shake my head at the irony of it.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#14

Post by gregd »

ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

"The backup HD" doesn't necessarily equate to "that means you had a RAID setup". How did you arrive at that conclusion? Budget hosts are notorious for having little to no backup solutions in place simply because, well I don't really know why? Costs? Complexity?

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#15

Post by JPKab »

It's noble of you to come clean and own your mistake, but let me say this over and over:

You should never, ever provide an environment that stores people's hard work without having professionals who know how to safeguard it.

If it makes you feel any better, I recently had to clean up a mess in a huge enterprise IT shop, (if I were to name the organization you would immediately know them) involving hundreds of thousands of man-hours of work lost due to a lazy, incompetent DBA and the clueless management above her.

This "DBA" was the kind of person who came in at 9:45AM, took a 2 hour lunch at noon, and left at 3:30. Did I mention she refused a work from home option?

She didn't know how to do chron jobs, so all of her backup scripts had to be run manually. If she was on vacation, they didn't get run. Surprise Surprise, the DB died after her long pre-Christmas vacation. Zero backups for the first 3 weeks of December.

Even "professionals" can be suspect sometimes.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#16

Post by gregd »

jrussbowman wrote: post #13
gregd wrote: post #2

"We need to get programming talent on-board." Sounds to me like they still haven't learned their lesson...

glad I wasn't the only one who had that as a first thought. as an ops guys I wasn't sure if I should be offended or just shake my head at the irony of it.

Well apparently someone didn't like my comment since it got down-voted. Whatever. As someone with a background in systems administration, it bothers me when this profession gets left out of the equation, far too often.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#18

Post by nwilkens »

I see this too many times.. and have read about this more than once on HN in recent memory.

Hire a proper system administration company early to work with you on these types of things. There are many companies out there that do this. I happen to run a company that does this, so I know that you can add an expert admin to your team for $100-200/mo.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#19

Post by ashray »

gregd wrote: post #14
ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

"The backup HD" doesn't necessarily equate to "that means you had a RAID setup". How did you arrive at that conclusion? Budget hosts are notorious for having little to no backup solutions in place simply because, well I don't really know why? Costs? Complexity?

The author said that the primary hard disk failed and the host had told him there was a backup hard disk that hummed along for a year.

I cannot imagine any other practical situation in which a 'backup hard disk' would automatically kick in - apart from a RAID setup.

I know that budget hosts do not backup data off site, but they do tend to maintain their hardware, RAID arrays, etc.

However, I admit that I am all too unfamiliar with shared hosting environments in today's day and age, it's either a cloud or dedicated server for me - and for my budget dedicated servers, hosts have been pretty proactive about replacing bad hard drives.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#20

Post by jrussbowman »

gregd wrote: post #16
jrussbowman wrote: post #13

Earlier quoted context omitted.

glad I wasn't the only one who had that as a first thought. as an ops guys I wasn't sure if I should be offended or just shake my head at the irony of it.

Well apparently someone didn't like my comment since it got down-voted. Whatever. As someone with a background in systems administration, it bothers me when this profession gets left out of the equation, far too often.

judging by some of the other comments I think people believe sysadmins can and have been replaced by heroku, aws and such.
Post reply on HN