20+ hour outage due to EC2/EBS on BitBucket

#1

Post by zx » Sat, Oct 03, 2009, 7:30 PM UTC

20+ hour outage due to EC2/EBS on BitBucket
bitbucket.org

Re: 20+ hour outage due to EC2/EBS on BitBucket

#2

Post by rogerthat » Sat, Oct 03, 2009, 8:57 PM UTC

Despite this, Bitbucket is great. Private repository with a free account option - you don't get that on GitHub.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#3

Post by shizcakes » Sat, Oct 03, 2009, 9:02 PM UTC

Ugh. I am working on the next gen architecture for our site, and I wanted to focus on hosting it in a cloud - but all these outages give me no confidence that cloud-based hosting is really all that ready for primetime yet.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#4

Post by durana » Sat, Oct 03, 2009, 9:05 PM UTC

Everything fails. Design systems that minimize the impact failures have on your customers. Moving to another provider isn't going to fix the problem. Data like this should be stored in more than one place.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#5

Post by jespern » Sat, Oct 03, 2009, 9:10 PM UTC

I'm here to answer questions if there are any (I run Bitbucket.)

Re: 20+ hour outage due to EC2/EBS on BitBucket

#6

Post by wmf » Sat, Oct 03, 2009, 9:20 PM UTC

durana wrote: post #4

Everything fails. Design systems that minimize the impact failures have on your customers. Moving to another provider isn't going to fix the problem. Data like this should be stored in more than one place.

In reality, replicating across two clouds is difficult and expensive. It's quite possible that Bitbucket wouldn't exist at all if they had used such an architecture.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#7

Post by shizcakes » Sat, Oct 03, 2009, 9:24 PM UTC

jespern wrote: post #5

I'm here to answer questions if there are any (I run Bitbucket.)

What are you thinking in terms of movement / failover at this point?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#8

Post by neurotech1 » Sat, Oct 03, 2009, 9:31 PM UTC

shizcakes wrote: post #3

Ugh. I am working on the next gen architecture for our site, and I wanted to focus on hosting it in a cloud - but all these outages give me no confidence that cloud-based hosting is really all that ready for primetime yet.

IHMO AWS Windows instances may not be ready for prime time. This issue seems not to affect Linux based instances.

There are other cloud providers like the YC favorite, SliceHost - I have no direct experience with them but may soon try them out.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#9

Post by jespern » Sat, Oct 03, 2009, 9:33 PM UTC

shizcakes wrote: post #7

jespern wrote: post #5

I'm here to answer questions if there are any (I run Bitbucket.)

What are you thinking in terms of movement / failover at this point?

We've been contacted by several hosting providers, and right now, Rackspace seems pretty nice.

The problem isn't that we don't have failover here, it's that we store all repositories on a single EBS volume. This has worked great for us in the past, but as of last night, that volume has become virtually unavailable to us. It doesn't matter which instance we mount it on, the throughput we get from it is excruciating.

If, or at this point--when, we move, the disk architecture will look different, and general failover will be less of an issue.

Amazon has for the past 8-10 hours been investigating the issue, and we're left pretty dumbfounded as of to what has happened exactly. I'll summarize everything in a blog post once the chaos is over.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#10

Post by Kayem » Sat, Oct 03, 2009, 9:45 PM UTC

An unfortunate event, however, re-iterates to system admins why the cloud should only be used as a low tier of storage -- for now.