20+ hour outage due to EC2/EBS on BitBucket • Hacker News Board

Re: 20+ hour outage due to EC2/EBS on BitBucket

#21

Post by jespern » Sat, Oct 03, 2009, 10:55 PM UTC

keefe wrote: post #20

jespern wrote: post #17

Earlier quoted context omitted.

You're absolutely right that the cloud is not magic, but you do get some guarantees with EBS. From their website: "Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component." We don't keep the database on the same EBS, and we have segmented database traffic out to several EBS volumes (for WAL, etc.) That's not the issue.…

>The only way to keep persistent data on EC2 is using EBS, and right now, it doesn't work for us, at all. S3 should work too? Unless it was a global EBS failure, you should be able to restore from any backup to a new set of instances and stores, why doesn't that work?

...As data you can access as a filesystem. S3 is great, but pretending it's a filesystem is going to get you awful performance.

As I said, our data is not lost, we have snapshots and backups, it's sitting right there on the mount, we're just not getting any sort of acceptable throughput. New instances does not fix the problem.

Ironically, we were looking into having S3 as the backend for our data, for scalability/redundancy purposes, but this pretty much puts a stop to that.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#22

Post by gfodor » Sat, Oct 03, 2009, 11:03 PM UTC

jespern wrote: post #21

keefe wrote: post #20

Earlier quoted context omitted.

>The only way to keep persistent data on EC2 is using EBS, and right now, it doesn't work for us, at all. S3 should work too? Unless it was a global EBS failure, you should be able to restore from any backup to a new set of instances and stores, why doesn't that work?

...As data you can access as a filesystem. S3 is great, but pretending it's a filesystem is going to get you awful performance. As I said, our data is not lost, we have snapshots and backups, it's sitting right there on the mount, we're just not getting any sort of acceptable throughput. New instances does not fix the problem. Ironically, we were looking into having S3 as the backend for our data, for scalability/red…

When we had the problem we fixed it by snapshotting the screwed up volume, and creating a new volume from that snap. Did you guys try this?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#23

Post by gfodor » Sat, Oct 03, 2009, 11:06 PM UTC

jespern wrote: post #19

keefe wrote: post #16

Earlier quoted context omitted.

What do you think of the other post critcizing your architecture? I am doing my own arch work on EC2, so I am trying to understand exactly what caused this failure. Is this a problem with your instance being able to access any EBS? Why couldn't you spin up another instance with a fresh EBS from a backup and redirect DNS to that instance?

When this sort of thing happens, it's very easy to point out all the things you should've done differently. In retrospect, everything's easier. You can't anticipate everything, and as I've pointed out in another comment here, this one is rather exotic. Quick summary of what the problem is: We have an EBS volume. It mounts fine, appears fine. The problem is that it's excruciatingly slow. We can't serve data from the v…

Ok, that answers my other question. I think the fundamental issue (aside from the amazon issue) is you had bytes living on a single EBS disk that weren't replicated to another disk. For important data, this is probably a bad idea regardless of backup strategy, etc.

Edit: By the way, the point here isn't to say "you guys screwed up" but to underscore that these types of issues aren't 100% Amazon's fault either, both parties had issues and the tone of your post seems to be "EC2 and EBS are not reliable we are switching off of it" when the truth lies somewhere in the middle.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#24

Post by jespern » Sat, Oct 03, 2009, 11:07 PM UTC

jespern wrote: post #21

keefe wrote: post #20

Earlier quoted context omitted.

>The only way to keep persistent data on EC2 is using EBS, and right now, it doesn't work for us, at all. S3 should work too? Unless it was a global EBS failure, you should be able to restore from any backup to a new set of instances and stores, why doesn't that work?

...As data you can access as a filesystem. S3 is great, but pretending it's a filesystem is going to get you awful performance. As I said, our data is not lost, we have snapshots and backups, it's sitting right there on the mount, we're just not getting any sort of acceptable throughput. New instances does not fix the problem. Ironically, we were looking into having S3 as the backend for our data, for scalability/red…

(I don't know why I can't reply to the post below, so I'll reply to myself):

Yes, we did try this, and it produced the same problem.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#25

Post by moe » Sat, Oct 03, 2009, 11:16 PM UTC

Kayem wrote: post #18

moe wrote: post #15

Earlier quoted context omitted.

What you are saying makes no sense. Everything fails occassionally. Amazon probably has a team of highly specialized engineers on the task right now, working under the pressure of a few dozen disgruntled customers and under the eyes of worldwide press. Could your company respond with an equal intensity if this was your own hardware? Will your SAN supplier whip his staff on-site as fast as they will for amazon?

"Everything fails occassionally. Amazon probably ..." I wish I could use that line to explain to my company why our business has come to a complete halt. Granted, your business is completely web-based and in the cloud, which is why I specifically made a mention to system admins why the cloud is not reliable enough to be a high-level tier of storage yet. Why doesn't that make sense? I wasn't trying to offend you or yo…

I wish I could use that line to explain to my company why our business has come to a complete halt.

It's a completely valid and reasonable business decision.

For most companies the risk of amazon downtime is simply not a deciding factor when held against what it would cost to maintain an own datacenter with remotely similar properties.

a high-level tier of storage yet. Why doesn't that make sense?

I guess I didn't get what you mean by "high-level tier storage"? Most companies have at most two tiers: Live and Snapshot-Backup. If you're a bank or fortune XXX with truly multi-tiered storage then yes, your inhouse staff might be able to do it better. But it will probably cost quite a bit more than ec2 and the business case for that is imho rather the exception than the rule.

But we have control over our own hardware, so there's really no comparison.

Well, I think you overestimate your capabilities there (unless you are a fortune 500). Amazon doesn't face downtimes over disk or server failures - and neither would you. The real question is who can debug and resolve complicated failure modes faster (you know, nasty stuff, heisenbugs).

Not meaning to offend you either but my money would be on amazon. That's why I questioned your broad statement of "not a high-level tier storage". How much higher level than backed by a 50.000-servers operation can it get?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#26

Post by jespern » Sat, Oct 03, 2009, 11:18 PM UTC

gfodor wrote: post #23

jespern wrote: post #19

Earlier quoted context omitted.

When this sort of thing happens, it's very easy to point out all the things you should've done differently. In retrospect, everything's easier. You can't anticipate everything, and as I've pointed out in another comment here, this one is rather exotic. Quick summary of what the problem is: We have an EBS volume. It mounts fine, appears fine. The problem is that it's excruciatingly slow. We can't serve data from the v…

Ok, that answers my other question. I think the fundamental issue (aside from the amazon issue) is you had bytes living on a single EBS disk that weren't replicated to another disk. For important data, this is probably a bad idea regardless of backup strategy, etc. Edit: By the way, the point here isn't to say "you guys screwed up" but to underscore that these types of issues aren't 100% Amazon's fault either, both p…

I apologize if that's the tone I'm relaying. I guess I'm just frustrated due to the time it's taking to fix it.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#27

Post by gfodor » Sat, Oct 03, 2009, 11:36 PM UTC

jespern wrote: post #26

gfodor wrote: post #23

Earlier quoted context omitted.

Ok, that answers my other question. I think the fundamental issue (aside from the amazon issue) is you had bytes living on a single EBS disk that weren't replicated to another disk. For important data, this is probably a bad idea regardless of backup strategy, etc. Edit: By the way, the point here isn't to say "you guys screwed up" but to underscore that these types of issues aren't 100% Amazon's fault either, both p…

I apologize if that's the tone I'm relaying. I guess I'm just frustrated due to the time it's taking to fix it.

Yea, it sounds like you guys are pretty much at Amazon's mercy right now, which sucks. I think a better way to look at these types of things is "what could we do to prevent this from happening again" and make a post about EBS gotchas.

Since it sounds like you guys are talking about straight up file storage I'd guess a good option would be to set up an HDFS cluster and be smart about locality/replication to minimize the latency.

Edit: Oh, and one more thing is take a backup that doesn't involve EBS snapshots. Maybe biweekly dump the entire sucker to S3 or something or have it getting pushed there all the time. This is something we've been meaning to do since the snapshotting capabilities of EBS are still a bit too magical for me to sleep well at night. (To be fair though, they've worked great when we've needed them to.)

Re: 20+ hour outage due to EC2/EBS on BitBucket

#28

Post by durana » Sat, Oct 03, 2009, 11:45 PM UTC

wmf wrote: post #6

durana wrote: post #4

Everything fails. Design systems that minimize the impact failures have on your customers. Moving to another provider isn't going to fix the problem. Data like this should be stored in more than one place.

In reality, replicating across two clouds is difficult and expensive. It's quite possible that Bitbucket wouldn't exist at all if they had used such an architecture.

That's an over generalization. And difficult and expensive relative to what? When evaluating the cost of solutions to problems like this, the basic idea is to compare the cost of the solution and the cost of the problem (if realized) with the likelihood of the problem being realized factored in.

It's quite possible Bitbucket will cease to exist if they continue to have problems like this. I'm sure their customers are not happy. It does sound like they have plans to fix this once they get back up and running.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#29

Post by keefe » Sat, Oct 03, 2009, 11:48 PM UTC

jespern wrote: post #21

keefe wrote: post #20

Earlier quoted context omitted.

>The only way to keep persistent data on EC2 is using EBS, and right now, it doesn't work for us, at all. S3 should work too? Unless it was a global EBS failure, you should be able to restore from any backup to a new set of instances and stores, why doesn't that work?

...As data you can access as a filesystem. S3 is great, but pretending it's a filesystem is going to get you awful performance. As I said, our data is not lost, we have snapshots and backups, it's sitting right there on the mount, we're just not getting any sort of acceptable throughput. New instances does not fix the problem. Ironically, we were looking into having S3 as the backend for our data, for scalability/red…

Oh, I wasn't suggesting pretending it's a file system - I had been thinking of a place to dump the data for backups, thinking fresh instances + fresh EBS would solve the problem. I think you answered this already in the other post - that you booted a new instance and a new EBS with some backup and the problem remained?? This seems like such a horrendous failure on AWS' part, unless it has something to do with how you are accessing the EBS (too many connections or something). I could understand if a given EBS fails, but if you can restore the data from an independent backup and spin back up with new instances and new EBS this indicates a very concerning systemic problem in EBS!

Re: 20+ hour outage due to EC2/EBS on BitBucket

#30

Post by Kayem » Sat, Oct 03, 2009, 11:49 PM UTC

moe wrote: post #25

Kayem wrote: post #18

Earlier quoted context omitted.

"Everything fails occassionally. Amazon probably ..." I wish I could use that line to explain to my company why our business has come to a complete halt. Granted, your business is completely web-based and in the cloud, which is why I specifically made a mention to system admins why the cloud is not reliable enough to be a high-level tier of storage yet. Why doesn't that make sense? I wasn't trying to offend you or yo…

I wish I could use that line to explain to my company why our business has come to a complete halt. It's a completely valid and reasonable business decision. For most companies the risk of amazon downtime is simply not a deciding factor when held against what it would cost to maintain an own datacenter with remotely similar properties. a high-level tier of storage yet. Why doesn't that make sense? I guess I didn't ge…

I completely agree with your points. Again, I was speaking to people who manage their own SANs, and may be looking to use the Cloud as an additional tier of storage, with the same reliability as a local array. Reliable in the sense that they would never have to worry about Internet latency, network nodes going down, or anything else that they have absolutely no knowledge or control over and could potentially affect the performance/operation of an application, ultimately disabling me from meeting business requirements.

If there are no business requirements to meet, I have no arguments, the Cloud is where I'm at!