Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Amazon EBS sucks. I just lost all my data"

developer.amazonwebservices.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: "Amazon EBS sucks. I just lost all my data"

#51

Post by wmf »

wmf wrote: post #50
jasonkester wrote: post #44

This post happens about once a week on the Amazon forums. I've watched it play out dozens of times on the S3 and Cloudfront forums too, and every single time it turns out to be operator error. In this case, the guy didn't realize he needed to take snapshots of his volumes. It's not surprising, really, since the documentation isn't so great for AWS, and it's probably even more painful knowing that it would have been a…

OTOH, if operators keep making the same mistake over and over again, maybe the UI should be changed. I think there's a fundamental mismatch in the EC2 control panel because it looks like anyone should be able to use it, but you have to be a competent sysadmin to use it safely .

Reuven Cohen makes a more detailed version of this point: "[Amazon] expect a certain level of knowledge of both system administration as well as how AWS itself has been designed to be used. Newbies need not apply or should use at you're own risk. Which isn't all that clear to a new user, who hears that cloud computing is safe and the answer to all your problems. ... You need the late adopters for the real revenue opportunities, but these same late adopters require a different more gentle kind of cloud service... As IaaS matures it is becoming obvious that the "Über Geek" developers who first adopted the service is not where the long tail revenue opportunities are." http://www.elasticvapor.com/2010/05/failure-as-service.html

Re: "Amazon EBS sucks. I just lost all my data"

#53

Post by jemfinch »

patio11 wrote: post #27
brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

You can't outsource your liability. Of course you can. That is the entire reason the insurance industry exists. More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seem…

> More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own

EBS lets you create backup snapshots in S3 with one single command. The problem the (I think the epithet is warranted here) idiot who posted the original article had is that he didn't use it.

Re: "Amazon EBS sucks. I just lost all my data"

#54

Post by Tamerlin »

kiujhygfghjk wrote: post #22
mynameishere wrote: post #20

Earlier quoted context omitted.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!

What's more relevant is that the odds of either drive failing are identical. The likelihood that the first drive will fail doesn't change because the second one failed -- which is the classic statistical blunder. ("I got 42 heads in a row, the next one's BOUND to be tails!")

Every single drive in the datacenter typically has the same likelihood of failure (since they're usually the same make and model, and similar production runs), so the odds that a drive somewhere in a large data center is failing RIGHT NOW are in reality rather high.

The Big Mac ended up being a good example of this. The likelihood of having a memory error on a normal PC is low, because the likelihood of a single DIMM having a memory error is low.

Punch that up to the 22,000 or thereabouts DIMMs that populated the Big Mac cluster, and now you're looking at a very HIGH likelihood that a large-scale computation using the cluster will experience memory errors, and therefore produce invalid results -- which is why UVa ended up replacing the entire cluster with ECC-equipped machines in short order. Until they did that, researchers had to run simulations multiple times and compare results to make sure that their simulations weren't contaminated by memory errors.

Re: "Amazon EBS sucks. I just lost all my data"

#55

Post by jodrellblank »

shahin wrote: post #37
tkaemming wrote: post #32

Earlier quoted context omitted.

Saying it "doesn't mean crap" is an exaggeration. AWS's physical redundancy offers a marginal increase in reliability (roughly 3.5%, according to their stats) at potentially lower cost than providing the same system in-house because of economies of scale. They never promised to be 100% fail-safe (and would be foolish to do so.)

The point is that they are not backing up your data they way they advertise it. The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Also if adding physical redundancy improves reliability only by 3.5% it means that they have different definition of the term.

The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist.

Which bit of "multiple failures of the underlying hardware components" leads you to think that a single point of failure still exists?

Re: "Amazon EBS sucks. I just lost all my data"

#56

Post by mrtron »

patio11 wrote: post #27
brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

You can't outsource your liability. Of course you can. That is the entire reason the insurance industry exists. More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seem…

10 years later you realize you aren't covered for that cancer surgery because you didn't check off the box saying you were a smoker on the 100 page form.

Re: "Amazon EBS sucks. I just lost all my data"

#57

Post by vl »

Most people that don't have extended experience with large-scale data stores do not understand basic principle: redundancy decreases probability of data loss, but it never eliminates it completely. All massive data stores slowly bleed data, it's just they bleed it so slow that it's acceptable for most scenarios. In case of this specific example, once number of users is large enough, there always be somebody who lost their volume.

To illustrate this: think about a-la-GFS randomly triplicating data store on 1000 nodes. Once enough data is put in (lets say 100M blobs), there always be blob unique to any given triplet. In other words simultaneous loss of any 3 nodes out of 1000 will always result in data loss. (Simultaneous is in the sense "faster than time to detect failure and recover"). Of course failures are not limited to node loss, but there is corruption in transit, hard drive loss, bad sectors, rack-level failures. As the volume of the data and number of nodes grows it all adds up, so even if for each particular blob mean time to data loss is astronomically high, probability to loose some blob on any given day is very real.

Re: "Amazon EBS sucks. I just lost all my data"

#58

Post by shlomo »

mattew wrote: post #12

Is there any way to set EBS to auto-snapshot on a specified time period through the existing control panel interface? Are snapshots possible through the API?

This article includes a script that will perform a consistent snapshot of an XFS-formatted EBS drive (in linux).

http://alestic.com/2009/09/ec2-consistent-snapshot

Consistent snapshots are needed when the disk contains multiple files that must be kept in sync with each other - such as a database's data files and the journal. If your DB gets its disk snapshotted while these files are out of sync, the DB will need to recover the missing data by replaying the journal. Not all DB's can do this reliably for all types of changes (e.g. MySQL's MyISAM engine is fragile in this regard). So you need to make sure that the files in the snapshot all reflect a consistent state of the disk.

Re: "Amazon EBS sucks. I just lost all my data"

#59

Post by jacquesm »

brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

Amen.

It's quite amazing how many people think they can point the finger at some third party because they 'handed it off'. After all, if it's amazon then it's safe as the bank of England right?

But even banks can burn down and even amazon does not owe you a penny if they lose your stuff.

They'll say sorry, really nicely and maybe they'll offer you some store credit, comes in great when you have lots of free time, you can get some books. Maybe on how to back-up your data or so.

But if you are halfway competent you smile and take that backup and you continue your business, because the value of your business outweighs amazons liability by a huge factor.

Backing up is risk mitigation, if the risk to you is small then you can afford to live without backups, it means that the cost of backing up outweighed the cost of re-creation of the data.

If the risk is larger then you probably should accept that, and go do something about it.

Either way, the responsibility is yours.

Re: "Amazon EBS sucks. I just lost all my data"

#60

Post by jacquesm »

shahin wrote: post #26
brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

"You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!

AWS promises you physical redundancy, what they don't offer you is an ironclad guarantee that nobody, customer included will ever screw up. So you need backups. It's really simple.
Post reply on HN