Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Amazon EBS sucks. I just lost all my data"

developer.amazonwebservices.com

Post reply on HN
Print view View original

11–20 of 67 posts

Re: "Amazon EBS sucks. I just lost all my data"

#11

Post by mechanical_fish »

jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

There are people in the world who do not understand probability. Like the OP:

EBS is either reliable or not. You cannot be a little pregnant.

Even the OP's metaphor is broken! The saying "you cannot be a little bit pregnant" may be traditional, but it too is only an approximation. There are these things called miscarriages. They happen all the time, often before a woman realizes she is pregnant. Then there are the false pregnancies:

http://en.wikipedia.org/wiki/False_pregnancy

And these are just the common edge cases.

Re: "Amazon EBS sucks. I just lost all my data"

#13

Post by jrockway »

Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed.

Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; three drive failures in a week!)

My fatal mistake, BTW, was ordering from Newegg. Apparently they do not ship OEM drives correctly, and they are almost guaranteed to fail. I was a little suspicious when I saw a raw drive in a plastic shell with some packing peanuts around it. When I had the drives replaced, they did not come from the factory that way!

Re: "Amazon EBS sucks. I just lost all my data"

#14

Post by geedee77 »

brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

I agree with this 100%. It seems that an awful lot of companies like to pass the blame as soon as something unwanted happens. Yes, it may be the underlying fault of a service provider but, to your customers, you are the only company they deal with and it's your reputation that will be impacted no matter how many posts you put on your site and all over the internet.

Using service companies does not excuse poor customer service.

Re: "Amazon EBS sucks. I just lost all my data"

#15

Post by andrewvc »

brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

That's true, I agree 100%.

I'd add to that though, if you're a small company sometimes you must recognize that a larger 3rd party can use scale to provide a more reliable service than you can.

Re: "Amazon EBS sucks. I just lost all my data"

#17

Post by X-Istence »

jrockway wrote: post #13

Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed. Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; th…

I have bought hundreds of OEM drives from NewEgg, all of them have had the plastic shell around them, and so far I've seen a 2% failure rate.

The plastic shell is the way the manufacturers ship them direct to NewEgg, that is not NewEgg's doing.

Re: "Amazon EBS sucks. I just lost all my data"

#18

Post by blantonl »

mattew wrote: post #12

Is there any way to set EBS to auto-snapshot on a specified time period through the existing control panel interface? Are snapshots possible through the API?

there is not, however it is really easy to write a script to freeze the volume and execute a snapshot.

# xfs_freeze -f /data # ec2-create-snapshot vol-###### # xfs_freeze -u /data

Re: "Amazon EBS sucks. I just lost all my data"

#19

Post by bcl »

dogas wrote: post #6

Why is this on HN? AWS provides a great way to back up EBS volumes called snapshots. Snapshots only store the deltas from the previous snapshot, and all the work to create one is done by AWS, not the server it is attached to. This guy didn't read the docs and did not use AWS snapshots. It was the equivalent of not having a backup strategy for your local hard drive.

At first I thought the same thing. But I think this is a useful discussion to have on HN, since not everyone here understands probabilities or the realities of 'cloud' services.

Maybe by having this the top rated story for a bit it will make someone think twice about their virtual infrastructure and examine their assumptions for errors. Do it for the data ;)

Re: "Amazon EBS sucks. I just lost all my data"

#20

Post by mynameishere »

jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?
Post reply on HN