Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our Server’s Hard Drive is Dead. We didn’t have a backup.

blog.method.ac

Post reply on HN
Print view View original

21–30 of 88 posts

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#21

Post by dangrossman »

ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

When you buy unmanaged servers, the host isn't monitoring RAID health -- they don't have any remote access to your machine except maybe IPMI for reboots. I've rented servers from various providers for a decade and none has ever monitored my hard drives... plenty have failed, including disks in a RAID and RAID adapters themselves; they get replaced when I call up and tell someone the server won't boot and I need someone to go take a look.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#22

Post by jefe78 »

jrussbowman wrote: post #20
gregd wrote: post #16

Earlier quoted context omitted.

Well apparently someone didn't like my comment since it got down-voted. Whatever. As someone with a background in systems administration, it bothers me when this profession gets left out of the equation, far too often.

judging by some of the other comments I think people believe sysadmins can and have been replaced by heroku, aws and such.

It's a recurring theme on HN. I'm a sysadmin and cringe when reading these rather common posts.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#23

Post by gregd »

dangrossman wrote: post #21
ashray wrote: post #9

I don't understand, you had a backup HD - that means you had a RAID setup. Why didn't your host replace the damaged hard disk ? In my experience hosts usually monitor RAID health on their servers and if there is a problem they replace the bad hard drives at the quickest opportunity.. and I'm talking about budget hosts. EDIT: Too many to respond to below so just editing in here. The author mentioned that the primary h…

When you buy unmanaged servers, the host isn't monitoring RAID health -- they don't have any remote access to your machine except maybe IPMI for reboots. I've rented servers from various providers for a decade and none has ever monitored my hard drives... plenty have failed, including disks in a RAID and RAID adapters themselves; they get replaced when I call up and tell someone the server won't boot and I need someo…

Thank you. This was exactly what I was getting at. 1&1 is on the short list of budget hosts that are notorious for doing the bare minimum to get your money.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#24

Post by gregd »

jrussbowman wrote: post #20
gregd wrote: post #16

Earlier quoted context omitted.

Well apparently someone didn't like my comment since it got down-voted. Whatever. As someone with a background in systems administration, it bothers me when this profession gets left out of the equation, far too often.

judging by some of the other comments I think people believe sysadmins can and have been replaced by heroku, aws and such.

I would have to agree. But I would also have to agree that Heroku or AWS may have been a better choice than a 1&1 dedicated server in this particular instance.

However, Heroku and AWS are no substitute for good systems administration. They are good substitutes for data centers given signed and documented SLAs.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#25

Post by gregd »

tempestn wrote: post #5

"Technical support informed me that my first HD died 20 days into my contract. The backup HD hummed along for a year." That sounds more like RAID than a backup HD.

This doesn't sound like RAID at all. How are you guys coming to this conclusion? At best it sounds like a server with a master/slave hard drive setup...you know, something from the early 2000s.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#26

Post by jefe78 »

gregd wrote: post #8
jthol wrote: post #4

Earlier quoted context omitted.

What do you mean?

I mean that Systems Administration is as thoroughly exhausting a career path as programming. They didn't lose information because they coded something wrong or inadequately.

Perhaps it's me, but I don't see the roles of SA and Dev as interchangeable. Yet, HN seems to present a world where all Devs are SAs. As a SA, it's frustrating.

All of that to say; I agree with what you're saying. I just wonder how to get the message across that maybe both roles aren't the same...

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#27

Post by sc00ter »

Not to be depended on as a substitute for backup, but 'dead' doesn't necessarily mean dead. Forensic recovery (either DIY or professional, depending on the nature of the failure) may still be an option.

Logic board failures are common, and replacements cheap (the cost of a new HD of the same model), data can be highly recoverable from soft failures. Mechanical failure is the worst case, but as long as the platter(s) is/are in tact, not insurmountable.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#28

Post by verelo »

JPKab wrote: post #15

It's noble of you to come clean and own your mistake, but let me say this over and over: You should never, ever provide an environment that stores people's hard work without having professionals who know how to safeguard it. If it makes you feel any better, I recently had to clean up a mess in a huge enterprise IT shop, (if I were to name the organization you would immediately know them) involving hundreds of thousan…

Sounds like they need a technical founder. Stock well spent imho...

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#29

Post by Brandon0 »

nwilkens wrote: post #18

I see this too many times.. and have read about this more than once on HN in recent memory. Hire a proper system administration company early to work with you on these types of things. There are many companies out there that do this. I happen to run a company that does this, so I know that you can add an expert admin to your team for $100-200/mo.

That is actually surprisingly cheap. Care if I ask what types of services one would get at those rates?

You're absolutely right though, for a company like OP's, if they are so short on cash, it makes a lot of sense to get someone in even if just for the week to address these types of fundamental problems.

Re: Our Server’s Hard Drive is Dead. We didn’t have a backup.

#30

Post by Brandon0 »

gregd wrote: post #25
tempestn wrote: post #5

"Technical support informed me that my first HD died 20 days into my contract. The backup HD hummed along for a year." That sounds more like RAID than a backup HD.

This doesn't sound like RAID at all. How are you guys coming to this conclusion? At best it sounds like a server with a master/slave hard drive setup...you know, something from the early 2000s.

A master/slave hard drive setup would have data loss. If the main hard drive died and it wasn't a RAID, surely they would have noticed.
Post reply on HN