Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our Server’s Hard Drive is Dead. We didn’t have a backup.

blog.method.ac

Post reply on HN
Print view View original

31–40 of 88 posts

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#31

Post by femto »

Chances are that most of the information is still physically there, just that it is inaccessible. First thing I would do is physically obtain the drive, so even if it's inaccessible you have the information in your possession.

From your blog post, I'd assume you don't have the knowledge to attempt recovery yourself, so call in an expert to handle the data recovery for you. At this stage, it is a matter of what the information is worth to you, compared to the cost of recovery. Almost any intervention is possible, for a price.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#32

Post by underwater »

gregd wrote: post #25
tempestn wrote: post #5

"Technical support informed me that my first HD died 20 days into my contract. The backup HD hummed along for a year." That sounds more like RAID than a backup HD.

This doesn't sound like RAID at all. How are you guys coming to this conclusion? At best it sounds like a server with a master/slave hard drive setup...you know, something from the early 2000s.

Mirrored disks are RAID 1: http://en.wikipedia.org/wiki/Standard_RAID_levels

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#33

Post by ashray »

dangrossman wrote: post #21
ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

When you buy unmanaged servers, the host isn't monitoring RAID health -- they don't have any remote access to your machine except maybe IPMI for reboots. I've rented servers from various providers for a decade and none has ever monitored my hard drives... plenty have failed, including disks in a RAID and RAID adapters themselves; they get replaced when I call up and tell someone the server won't boot and I need someo…

You're right, I seem to have gotten lucky with my unmanaged hosting (3 times over with different hosts). They seem to have some sort of hardware interface to monitor RAID health, of course, this is hardware RAID so maybe that's where the setup differs. I was surprised when I received an email from them one morning about a year ago saying "Hey, one of your RAID drives failed so we replaced it, just FYI".

It's true that a RAID failure may go unnoticed by a sysadmin for a year or more if they don't have proper checks setup for themselves.

I guess the only thing that could've been done in this case was to have a backup cronjob or use a provider that takes care of this stuff..

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#35

Post by rincebrain »

jefe78 wrote: post #26
gregd wrote: post #8

Earlier quoted context omitted.

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

Perhaps it's me, but I don't see the roles of SA and Dev as interchangeable. Yet, HN seems to present a world where all Devs are SAs. As a SA, it's frustrating. All of that to say; I agree with what you're saying. I just wonder how to get the message across that maybe both roles aren't the same...

They're synergistic roles, yes, and ideally you would like one with a mix of both [either a dev with sufficient experience in system administration or a system administrator with sufficient experience in development] to tackle the messes that crop up, but I haven't seen anyone claiming they're interchangeable - just that most of the population of HN is some overlap of the two.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#36

Post by GalacticDomin8r »

ashray wrote: post #11
gregd wrote: post #8

Earlier quoted context omitted.

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

The most basic backup would be to have a remote backup of your files and databases. That's a simple shell script and cron job combination. I can understand that one may not get around to setting it up (I had a massive project and only started doing this after about 7 years of running it..) but it's definitely very important and doesn't need a ton of Sysadmin experience or setup time.

It also isn't a backup unless it's tested.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#37

Post by Sanddancer »

ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

Repeat after me: RAID is not a backup. It will mitigate certain drive failures, but is not a backup. Period, end of statement. Controllers will forget their info, OSes will eat their partition tables, and will otherwise ruin your data. If you're not backing up your stuff, to a completely separate system, preferably to a completely separate service, you will lose data. Period.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#38

Post by jthol »

gregd wrote: post #8
jthol wrote: post #4

Earlier quoted context omitted.

What do you mean?

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

Ah. Yeah you have a point but a good dev would at least suggest doing backups of some sort.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#39

Post by GalacticDomin8r »

ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

Repeat after me: RAID is not a backup.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#40

Post by nuclear_eclipse »

jefe78 wrote: post #26
gregd wrote: post #8

Earlier quoted context omitted.

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

Perhaps it's me, but I don't see the roles of SA and Dev as interchangeable. Yet, HN seems to present a world where all Devs are SAs. As a SA, it's frustrating. All of that to say; I agree with what you're saying. I just wonder how to get the message across that maybe both roles aren't the same...

They aren't interchangeable, but there is a significant amount of overlap in the position, especially when you begin operating at scale. The amount and type of automation required when you're dealing with thousands of servers demands a breadth of knowledge that spans both domains.
Post reply on HN