Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Amazon EBS sucks. I just lost all my data"

developer.amazonwebservices.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: "Amazon EBS sucks. I just lost all my data"

#31

Post by mynameishere »

kiujhygfghjk wrote: post #22
mynameishere wrote: post #20

Earlier quoted context omitted.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!

If amazon is advertising that single failures can't cause data losses, then a single exploding power supply (for instance) breaks that agreement.

Re: "Amazon EBS sucks. I just lost all my data"

#32

Post by tkaemming »

shahin wrote: post #26
brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

"You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!

Saying it "doesn't mean crap" is an exaggeration. AWS's physical redundancy offers a marginal increase in reliability (roughly 3.5%, according to their stats) at potentially lower cost than providing the same system in-house because of economies of scale. They never promised to be 100% fail-safe (and would be foolish to do so.)

Re: "Amazon EBS sucks. I just lost all my data"

#33

Post by astrodust »

mechanical_fish wrote: post #11
jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

There are people in the world who do not understand probability. Like the OP: EBS is either reliable or not. You cannot be a little pregnant. Even the OP's metaphor is broken! The saying "you cannot be a little bit pregnant" may be traditional, but it too is only an approximation. There are these things called miscarriages . They happen all the time, often before a woman realizes she is pregnant. Then there are the f…

You can be "a little bit pregnant" when you're talking about large groups of people. Of the women in your country, how many are pregnant at any given time? It's probably "a little bit". Obviously someone is confusing a statistical sampling of 1 with the kind of volume Amazon must deal with.

I find it informative that a rare event such as this is being given so much press. If this was more routine, this wouldn't be news. Amazon seems to be doing better than 0.1% based on that alone.

Re: "Amazon EBS sucks. I just lost all my data"

#34

Post by jonknee »

mynameishere wrote: post #20
jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

The percentages they give aren't for a single drive failure, they're for data loss (multiple drive failure). So the odds are .1-.5%.

Re: "Amazon EBS sucks. I just lost all my data"

#36

Post by holdenc »

braindead_in wrote: post #30

Can this also happen to an S3 bucket? How do I backup an S3 bucket? Any ideas?

Also curious to know of any S3 bucket failure stories out there. I would assume this is uncommon, but I also assumed that the EBS failure rate was much lower prior to reading this.

Re: "Amazon EBS sucks. I just lost all my data"

#37

Post by shahin »

tkaemming wrote: post #32
shahin wrote: post #26

Earlier quoted context omitted.

"You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!

Saying it "doesn't mean crap" is an exaggeration. AWS's physical redundancy offers a marginal increase in reliability (roughly 3.5%, according to their stats) at potentially lower cost than providing the same system in-house because of economies of scale. They never promised to be 100% fail-safe (and would be foolish to do so.)

The point is that they are not backing up your data they way they advertise it. The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Also if adding physical redundancy improves reliability only by 3.5% it means that they have different definition of the term.

Re: "Amazon EBS sucks. I just lost all my data"

#38

Post by garnaat »

garnaat wrote: post #35
braindead_in wrote: post #30

Can this also happen to an S3 bucket? How do I backup an S3 bucket? Any ideas?

I suppose it's possible but S3 is designed for nine 9's wrt to durability. There are > 100 billion objects and I'm not aware of any being lost due to AWS fault.

Actually, I'm wrong. It's eleven 9's. http://bit.ly/ageV9D

Re: "Amazon EBS sucks. I just lost all my data"

#39

Post by mattew »

blantonl wrote: post #18
mattew wrote: post #12

Is there any way to set EBS to auto-snapshot on a specified time period through the existing control panel interface? Are snapshots possible through the API?

there is not, however it is really easy to write a script to freeze the volume and execute a snapshot. # xfs_freeze -f /data # ec2-create-snapshot vol-###### # xfs_freeze -u /data

One of my clients has 2 windows instances. I will need to look at the windows tools to see if there is something equivalent there.

Re: "Amazon EBS sucks. I just lost all my data"

#40

Post by btilly »

mynameishere wrote: post #20
jwr wrote: post #8

> "expect an annual failure rate (AFR) of between 0.1% –0.5%, where failure refers to a complete loss of the volume" Well, I think the OP has just experienced a sample from a probability distribution characterized above.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Your assumptions are unreasonable, insufficiently well specified, and are asking the wrong question.

If in a year out of a million hard drives only 5000 die, then you're projecting a 200 year average lifetime per disk drive. No real disk has that. A more reasonable 5 year average lifespan gives you 200,000 failures per year. Which is much worse.

Next, you're asking about the odds of 2 failing on the same machine. How many disks are on a machine? 1? 10? 100? Are failures independent events? It makes a huge difference. In fact they are not independent because when the motherboard craps out you lose access to all disks on that machine at once. At their scale it is too much work to figure out whether some of that data is recoverable - you just assume there is another copy somewhere and throw away the stale data. If you're wrong, then oops.

You are also throwing out the 15 minute disk replacement time. It may take 15 minutes to replace a disk, but that figure is irrelevant. To replace a disk you have to locate the machine, and it has to matter enough to you to send a person out. I guarantee you that the time before a person gets involved is going to average more than 15 minutes. Generally a lot more than 15 minutes. (Google famously takes the attitude that it is generally more work than it is worth to find the broken machine, and lets most dead machines sit there indefinitely. I wouldn't be surprised if other cloud providers imitate this.)

Next you have to consider that the end user shouldn't care about machines. For the purpose of redundancy Amazon is not going to keep multiple copies of the same data on the same machine. They are going to put them in different machines, and hopefully in different places. That will reduce the odds of a single failure losing your data.

All of that said I am somewhat shocked that Amazon would advertise a 0.5-0.1% rate of data loss as acceptable. I don't know Google's actual failure rate, but I'd be willing to bet large amounts of money that it is much lower than that.

For instance search for "gmail lost data". The only significant gmail data loss that turns up was in 2006. (See http://techcrunch.com/2006/12/28/gmail-disaster-reports-of-m... for more.) A grand total of 60 accounts got wiped out. Subsequently most of the lost data was restored from backup. (I doubt that the error was at the data storage layer.)

That's not just better than what Amazon delivers. That is ridiculously better.

Post reply on HN