"Amazon EBS sucks. I just lost all my data" • Hacker News Board

Re: "Amazon EBS sucks. I just lost all my data"

#41

Post by brk » Mon, May 10, 2010, 3:53 PM UTC

patio11 wrote: post #27

brk wrote: post #2

Although I sometimes get downvoted for this, I'll say it again: You can't outsource your liability. If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a cert…

You can't outsource your liability. Of course you can. That is the entire reason the insurance industry exists. More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seem…

What is your "insurance" for hosting? I'm not aware of any SLAs that actually pay out anything near equal value to an outage they caused. If you're paying a hosting provider $500/mo for a sever that you generate an average of $1000/hr in web sales from and they have a 3 hour outage, you'll get an SLA credit for a couple of bucks applied to your next months service.

The Internet is filled with news about hosting provider provided backups being unusable for a number of reasons at inopportune times.

You should have a backup copy of your code/databases in your control, on a machine that is completely independent from whatever you are doing your production hosting on. You should have done a "warm metal" install and test of that code another server to be sure that you can recover operations in a reasonable amount of time (whatever is appropriate for your case).

For your scale (based on your posts here, my assumption: Single developer, or developer with a couple of contractors; production site; 1-5K visits per month; non time-sensitive/mission-critical service.) you probably don't need high-availability auto-failover. But, you SHOULD have your DNS hosted separate from your hosting provider, you SHOULD have low TTL's, you SHOULD have a backup server in a warm state at some other provider, and you SHOULD know how to at least do a basic DNS update to redirect traffic over to a backup site that either runs the service or puts up a basic, friendly "OOPS, BRB" page.

I've often thought of a startup that would basically human-automate these things for guys like you. You still wouldn't be 100% self-sufficient, but you would be able to outsource SOME of your entire reliance on 1 provider. You'd then have to have 2 tiers of total failure (your provider, and this service) to encounter complete down time.

Re: "Amazon EBS sucks. I just lost all my data"

#42

Post by azim » Mon, May 10, 2010, 4:12 PM UTC

mynameishere wrote: post #31

kiujhygfghjk wrote: post #22

Earlier quoted context omitted.

Lots of people who didn't fail statistics assume that events are independant. A bit like, the chance of my machine catching fire are 1 in a 1000, my machine did catch fire and I lost both drives - the odds on that happening must be 1 in a million!

If amazon is advertising that single failures can't cause data losses, then a single exploding power supply (for instance) breaks that agreement.

If by "exploding" power supply you mean the potential for a power supply to explode with eruptive force and destroy other equipment in the rack, I believe that the other equipment being destroyed would be considered an multiple failure. It's very unlikely that a power supply could do that (although I guess anything is possible). A proper datacenter built with Datacenter-grade equipment will have multiple redundant power supplies in a blade enclosure, fed by different rails, which come off of different main lines in to the building. So a single failed source of power won't cause a failure.

Re: "Amazon EBS sucks. I just lost all my data"

#43

Post by skorgu » Mon, May 10, 2010, 4:45 PM UTC

btilly wrote: post #40

mynameishere wrote: post #20

Earlier quoted context omitted.

I failed statistics. If out of a million hard drives 5000 die in a year and take 15 minutes to swap, what are the odds of 2 failing on the same machine?

Your assumptions are unreasonable, insufficiently well specified, and are asking the wrong question. If in a year out of a million hard drives only 5000 die, then you're projecting a 200 year average lifetime per disk drive. No real disk has that. A more reasonable 5 year average lifespan gives you 200,000 failures per year. Which is much worse. Next, you're asking about the odds of 2 failing on the same machine. How…

To be fair you'd have to include failures of EBS snapshots and of failures to multiple datacenters to gain parity with Google in your comparison. I'm sure the gmail app doesn't use its storage subsystem naively no matter what the numbers are. You're absolutely right in general though.

Re: "Amazon EBS sucks. I just lost all my data"

#44

Post by jasonkester » Mon, May 10, 2010, 4:47 PM UTC

This post happens about once a week on the Amazon forums. I've watched it play out dozens of times on the S3 and Cloudfront forums too, and every single time it turns out to be operator error.

In this case, the guy didn't realize he needed to take snapshots of his volumes. It's not surprising, really, since the documentation isn't so great for AWS, and it's probably even more painful knowing that it would have been a single button click to back up his volume using Amazon's tools.

But in the end, there's nothing to see here. Just like the guy who wakes up in the morning to find all his S3 files mysteriously gone (after he 'renamed' his bucked the previous night by dropping and recreating it), it always turns out to be the user shooting himself in the foot.

And in the cases when Amazon actually does something wrong, they're always on top of it immediately and back with a public explanation within hours. (from my experience)

Re: "Amazon EBS sucks. I just lost all my data"

#45

Post by jrockway » Mon, May 10, 2010, 5:05 PM UTC

barrkel wrote: post #21

jrockway wrote: post #13

Hard drives suck just as bad. I have a RAID-1 built from three disks out of separate batches. Somehow, I wasn't paying attention to bad sectors the RAID software couldn't fix, and all the disks failed. Cheap 1TB disks and cheap cloud storage like EBS means that it's now cheaper than ever to lose a shit-ton of data. (I didn't actually lose anything important, the corrupted areas were not important files. But still; th…

Drives generally come packed either in those plastic shells, or in anti-static wrap, and then packed into specially cut foam (foam with slots, like a very parallel toaster) in a box with foam at the bottom and foam at the top. If you're not buying the drives in bulk, you don't get the box with foam.

Yeah, when I get the replacements from the drive manufacturer, that's how I receive them. But I've ordered pillows that are better packed than what I get from Newegg.

Re: "Amazon EBS sucks. I just lost all my data"

#46

Post by mark_l_watson » Mon, May 10, 2010, 5:34 PM UTC

Sounds like he did not make S3 snapshots of his EBS volumes. Ouch. I feel very confident about the robustness of data that I store on AWS because I can make an S3 snapshot, and recover from that snapshot on a fresh EC2 to test the backup. BTW, I changed the way I use AWS: now I always make bootable EBS images, increasing the size > the 10 GB limit so I snapshot my OS setup and data and apps all at the same time.

Re: "Amazon EBS sucks. I just lost all my data"

#47

Post by btilly » Mon, May 10, 2010, 5:42 PM UTC

skorgu wrote: post #43

btilly wrote: post #40

Earlier quoted context omitted.

Your assumptions are unreasonable, insufficiently well specified, and are asking the wrong question. If in a year out of a million hard drives only 5000 die, then you're projecting a 200 year average lifetime per disk drive. No real disk has that. A more reasonable 5 year average lifespan gives you 200,000 failures per year. Which is much worse. Next, you're asking about the odds of 2 failing on the same machine. How…

To be fair you'd have to include failures of EBS snapshots and of failures to multiple datacenters to gain parity with Google in your comparison. I'm sure the gmail app doesn't use its storage subsystem naively no matter what the numbers are. You're absolutely right in general though.

I'm positive that there is nothing naive in how Gmail uses storage. However I suspect that they are using standard best practices that are common throughout Google.

Re: "Amazon EBS sucks. I just lost all my data"

#48

Post by cemerick » Mon, May 10, 2010, 5:54 PM UTC

dogas wrote: post #6

Why is this on HN? AWS provides a great way to back up EBS volumes called snapshots. Snapshots only store the deltas from the previous snapshot, and all the work to create one is done by AWS, not the server it is attached to. This guy didn't read the docs and did not use AWS snapshots. It was the equivalent of not having a backup strategy for your local hard drive.

I submitted it because stories like this are common, but the underlying issues and misunderstandings aren't often aired out as they have been nicely here. Lots of people don't understand what they are and are not getting with "cloud" providers, nevermind grokking specifics like MTBF stats (and similar).

Re: "Amazon EBS sucks. I just lost all my data"

#49

Post by spudlyo » Mon, May 10, 2010, 5:57 PM UTC

blantonl wrote: post #18

mattew wrote: post #12

Is there any way to set EBS to auto-snapshot on a specified time period through the existing control panel interface? Are snapshots possible through the API?

there is not, however it is really easy to write a script to freeze the volume and execute a snapshot. # xfs_freeze -f /data # ec2-create-snapshot vol-###### # xfs_freeze -u /data

So that's an xfs feature, but is is this really necessary on a journaling filesystem? I guess for ext3 I'd replace ext_freeze with sync.

Re: "Amazon EBS sucks. I just lost all my data"

#50

Post by wmf » Mon, May 10, 2010, 6:16 PM UTC

jasonkester wrote: post #44

This post happens about once a week on the Amazon forums. I've watched it play out dozens of times on the S3 and Cloudfront forums too, and every single time it turns out to be operator error. In this case, the guy didn't realize he needed to take snapshots of his volumes. It's not surprising, really, since the documentation isn't so great for AWS, and it's probably even more painful knowing that it would have been a…

OTOH, if operators keep making the same mistake over and over again, maybe the UI should be changed. I think there's a fundamental mismatch in the EC2 control panel because it looks like anyone should be able to use it, but you have to be a competent sysadmin to use it safely.