Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Amazon EBS sucks. I just lost all my data"

developer.amazonwebservices.com

Post reply on HN
Print view View original

21–30 of 67 posts

Re: "Amazon EBS sucks. I just lost all my data"

#21

Post by barrkel »

jrockway wrote: post #13

Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed. Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; th…

Drives generally come packed either in those plastic shells, or in anti-static wrap, and then packed into specially cut foam (foam with slots, like a very parallel toaster) in a box with foam at the bottom and foam at the top. If you're not buying the drives in bulk, you don't get the box with foam.

Re: "Amazon EBS sucks. I just lost all my data"

#22

Post by kiujhygfghjk »

mynameishere wrote: post #20
jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!

Re: "Amazon EBS sucks. I just lost all my data"

#23

Post by lallysingh »

mynameishere wrote: post #20
jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Higher than expected: http://labs.google.com/papers/disk_failures.pdf

Re: "Amazon EBS sucks. I just lost all my data"

#25

Post by hga »

X-Istence wrote: post #17
jrockway wrote: post #13

Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed. Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; th…

I have bought hundreds of OEM drives from NewEgg, all of them have had the plastic shell around them, and so far I've seen a 2% failure rate. The plastic shell is the way the manufacturers ship them direct to NewEgg, that is not NewEgg's doing.

No, the plastic shelled disks are packed in solid foam in boxes holding something like 20 disks. Companies like NewEgg and Amazon break those boxes but do not follow the manufacturer's requirements (e.g. for shipping a failed disk back to them) when they repack them for individual orders.

Last time I bought disks ZipZoomFly properly repacked them by putting the shelled disks into individual foam boxes.

Re: "Amazon EBS sucks. I just lost all my data"

#26

Post by shahin »

brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

"You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!

Re: "Amazon EBS sucks. I just lost all my data"

#27

Post by patio11 »

brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

You can't outsource your liability.

Of course you can. That is the entire reason the insurance industry exists.

More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seems to get discovered that the most inconvenient of times.)

Re: "Amazon EBS sucks. I just lost all my data"

#28

Post by lamby »

mynameishere wrote: post #20
jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Very low. But that assumes they are independent events, which has never been the case in my experience.

Re: "Amazon EBS sucks. I just lost all my data"

#29

Post by hga »

geedee77 wrote: post #14
brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

I agree with this 100%. It seems that an awful lot of companies like to pass the blame as soon as something unwanted happens. Yes, it may be the underlying fault of a service provider but, to your customers, you are the only company they deal with and it's your reputation that will be impacted no matter how many posts you put on your site and all over the internet. Using service companies does not excuse poor custome…

This could well turn out to be a fatal flaw for enterprise adoption of third party clouds: when corporate IT staff screw up like this or just messing with a control panel (check the AWS forums where too often a "It disappeared!" complaint has a reply of "It was deleted at thus and so time and date using the control panel."), are they going to accept the blame or shift it to the third party?
Post reply on HN