Live data from Hacker News

20+ hour outage due to EC2/EBS on BitBucket

bitbucket.org

31–40 of 48 posts

Re: 20+ hour outage due to EC2/EBS on BitBucket

#31
post #23
post #19

Earlier quoted context omitted.

When this sort of thing happens, it's very easy to point out all the things you should've done differently. In retrospect, everything's easier. You can't anticipate everything, and as I've pointed out in another comment here, this one is rather exotic. Quick summary of what the problem is: We have an EBS volume. It mounts fine, appears fine. The problem is that it's excruciatingly slow. We can't serve data from the v…

Ok, that answers my other question. I think the fundamental issue (aside from the amazon issue) is you had bytes living on a single EBS disk that weren't replicated to another disk. For important data, this is probably a bad idea regardless of backup strategy, etc. Edit: By the way, the point here isn't to say "you guys screwed up" but to underscore that these types of issues aren't 100% Amazon's fault either, both p…

I don't think that this was a replication issue based on this comment :

>Booting a new instance, re-creating the volume from a >recent snapshot, doesn't help. The exact same problem >persists. Why? We don't know. Amazon's figuring it out.

If you can recreate the volume from a snapshot and hit it with fresh instances and run into the same problem, this is quite worrying. If it had resolved after a restore from backup, I would have felt better about EBS. As I see it there are only these options :

1) There is a general, systemic failure in EBS. You ran into it and highlighted it to AWS and they are fixing some problem. If other people are not having the same problem as you, I would be more inclined to think of #2.

2) Some usage pattern violates an assumption that was made when EBS was designed and screws it. Restoring from the backup reproduces the usage pattern. This could be simultaneous connections or # of distinct files in the volume, for example. One way to test this would be to split the data in the drive into a larger number of smaller EBS-es (EBSii? whatever the plural(: ) or throttle the simultaneous connections and see what happens.

did I miss anything?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#32

Despite this, Bitbucket is great. Private repository with a free account option - you don't get that on GitHub.

OTOH, since Github has some of my money, they have a bit more of an obligation (and incentive) to keep their servers up. Free services come and go as the owner pleases. (Hello, ma.gnol.ia.)

Re: 20+ hour outage due to EC2/EBS on BitBucket

#33

Ugh. I am working on the next gen architecture for our site, and I wanted to focus on hosting it in a cloud - but all these outages give me no confidence that cloud-based hosting is really all that ready for primetime yet.

IHMO AWS Windows instances may not be ready for prime time. This issue seems not to affect Linux based instances. There are other cloud providers like the YC favorite, SliceHost - I have no direct experience with them but may soon try them out.

Slicehost is not really a "cloud provider". You buy a VPS and use it "forever".

Re: 20+ hour outage due to EC2/EBS on BitBucket

#34
post #10

An unfortunate event, however, re-iterates to system admins why the cloud should only be used as a low tier of storage -- for now.

Even Google is down from time to time.

Basically, Bitbucket is having some downtime because of Amazon. But if it wasn't Amazon, it could be something else -- failing hardware, earthquakes, disgruntled datacenter employees, whatever. At least the Bitbucket folks can theoretically sit back and relax while Someone Else fixes this (rare) hardware problem.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#36
post #6

Earlier quoted context omitted.

In reality, replicating across two clouds is difficult and expensive. It's quite possible that Bitbucket wouldn't exist at all if they had used such an architecture.

I'm looking at cassandra as a possible solution for things like this.

Seems like a popular theme right now -- one of the github guys is working on a cassandra git backend at http://github.com/schacon/agitmemnon, and paul querna of ASF infrastructure is looking at doing the same for svn.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#37

Earlier quoted context omitted.

IHMO AWS Windows instances may not be ready for prime time. This issue seems not to affect Linux based instances. There are other cloud providers like the YC favorite, SliceHost - I have no direct experience with them but may soon try them out.

Slicehost is not really a "cloud provider". You buy a VPS and use it "forever".

Well, it's both. Rackspace Cloud Servers (basically, Slicehost post-acquisition) has an EC2-like api to spin up and down servers on demand, billed by the hour.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#39
post #10

An unfortunate event, however, re-iterates to system admins why the cloud should only be used as a low tier of storage -- for now.

To be fair, cloud storage is a huge step up in terms of reliability for small to medium size startups where the capital necessary to roll out their own hardware and people necessary to maintain that hardware is not within their budget. A lot of startups resort to very risky setups because they can't afford increased reliability, and Amazon and Rackspace seem to solve that problem. For a larger company, it makes more sense to expend capital on reliability because they can afford it and they want to hold the ax when something bad occurs.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#40

Despite this, Bitbucket is great. Private repository with a free account option - you don't get that on GitHub.

OTOH, since Github has some of my money, they have a bit more of an obligation (and incentive) to keep their servers up. Free services come and go as the owner pleases. (Hello, ma.gnol.ia.)

Not really - Bitbucket have gotten quite some of my money.

And yes I am considering chancing that, but mostly because I can't find a good mercurial client for windows.

Post reply on HN