Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

20+ hour outage due to EC2/EBS on BitBucket

bitbucket.org

Post reply on HN
Print view View original

41–48 of 48 posts

Re: 20+ hour outage due to EC2/EBS on BitBucket

#41

Post by jespern »

jespern wrote: post #5

I'm here to answer questions if there are any (I run Bitbucket.)

It was fixed around 4am (GMT+2) last night, with the assistance of Amazon. I'm just going to summarize what happened here:

We were attacked. Massive UDP DDOS. The flood of traffic prevented us from accessing our EBS store with any acceptable speeds, which is what caused everyone to think the problem was between our EC2 and the EBS. Of course this also explains why booting up a new instance and EBS didn't help anything.

Also, it's happening again now, and we're working with Amazon to remedy it once more.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#42

Post by tlrobinson »

jespern wrote: post #41
jespern wrote: post #5

I'm here to answer questions if there are any (I run Bitbucket.)

It was fixed around 4am (GMT+2) last night, with the assistance of Amazon. I'm just going to summarize what happened here: We were attacked. Massive UDP DDOS. The flood of traffic prevented us from accessing our EBS store with any acceptable speeds, which is what caused everyone to think the problem was between our EC2 and the EBS. Of course this also explains why booting up a new instance and EBS didn't help anythin…

Is there anything Amazon could have done to prevent this (or at least made diagnosing it easier), or is it a problem with your particular application?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#43

Post by jespern »

tlrobinson wrote: post #42
jespern wrote: post #41

Earlier quoted context omitted.

It was fixed around 4am (GMT+2) last night, with the assistance of Amazon. I'm just going to summarize what happened here: We were attacked. Massive UDP DDOS. The flood of traffic prevented us from accessing our EBS store with any acceptable speeds, which is what caused everyone to think the problem was between our EC2 and the EBS. Of course this also explains why booting up a new instance and EBS didn't help anythin…

Is there anything Amazon could have done to prevent this (or at least made diagnosing it easier), or is it a problem with your particular application?

We're talking UDP flood here, saturating our bandwidth. It never reached our servers, it just ate all the bandwidth on our connection. I guess what Amazon could have done is be quicker in spotting the DDOS and take measures to prevent it.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#44

Post by idlewords »

keefe wrote: post #31
gfodor wrote: post #23

Earlier quoted context omitted.

Ok, that answers my other question. I think the fundamental issue (aside from the amazon issue) is you had bytes living on a single EBS disk that weren't replicated to another disk. For important data, this is probably a bad idea regardless of backup strategy, etc. Edit: By the way, the point here isn't to say "you guys screwed up" but to underscore that these types of issues aren't 100% Amazon's fault either, both p…

I don't think that this was a replication issue based on this comment : >Booting a new instance, re-creating the volume from a >recent snapshot, doesn't help. The exact same problem >persists. Why? We don't know. Amazon's figuring it out. If you can recreate the volume from a snapshot and hit it with fresh instances and run into the same problem, this is quite worrying. If it had resolved after a restore from backup,…

Yes, the actual cause, a problem on the wire between them and Amazon. UDP flood eating up available bandwidth.

I didn't guess it either :-)

Re: 20+ hour outage due to EC2/EBS on BitBucket

#45

Post by spudlyo »

jespern wrote: post #43
tlrobinson wrote: post #42

Earlier quoted context omitted.

Is there anything Amazon could have done to prevent this (or at least made diagnosing it easier), or is it a problem with your particular application?

We're talking UDP flood here, saturating our bandwidth. It never reached our servers, it just ate all the bandwidth on our connection. I guess what Amazon could have done is be quicker in spotting the DDOS and take measures to prevent it.

So you never saw any evidence of this DDOS yourself? I'm somewhat skeptical of this explanation. It seems to me with shared infrastructure it'd be difficult to saturate just one customer's connection. It also doesn't make sense to me that this could be done without the traffic ever reaching your server. You used the phrases "our bandwidth" and "our connection" do things really work this way on the AWS cloud?

Anyway, I'm really sorry you guys had to go through all of this, and I hope whatever it is that caused it is fixed.

Re: 20+ hour outage due to EC2/EBS on BitBucket

#46

Post by keefe »

idlewords wrote: post #44
keefe wrote: post #31

Earlier quoted context omitted.

I don't think that this was a replication issue based on this comment : >Booting a new instance, re-creating the volume from a >recent snapshot, doesn't help. The exact same problem >persists. Why? We don't know. Amazon's figuring it out. If you can recreate the volume from a snapshot and hit it with fresh instances and run into the same problem, this is quite worrying. If it had resolved after a restore from backup,…

Yes, the actual cause, a problem on the wire between them and Amazon. UDP flood eating up available bandwidth. I didn't guess it either :-)

So, do you think setting a smaller max size on your EBS instances would have avoided this by spreading the traffic, so if you were using 1TB using 10 100GB ones instead and federating queries across them?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#47

Post by tlrobinson »

jespern wrote: post #43
tlrobinson wrote: post #42

Earlier quoted context omitted.

Is there anything Amazon could have done to prevent this (or at least made diagnosing it easier), or is it a problem with your particular application?

We're talking UDP flood here, saturating our bandwidth. It never reached our servers, it just ate all the bandwidth on our connection. I guess what Amazon could have done is be quicker in spotting the DDOS and take measures to prevent it.

So it was actually entirely unrelated to EBS? The reason it was taking 10 seconds to do an "ls" was simply a saturated connection to your server, not too much EBS activity?

Re: 20+ hour outage due to EC2/EBS on BitBucket

#48

Post by tve »

From what I can piece together, it seems the real problem isn't EBS, it's that the security groups are implemented at the host level (the machine on which your instance runs). This means that the UDP flood reached your host where it got dropped due to the security group rules, but it still had a performance impact, on EBS in your case, just because of the sheer volume of packets. The trouble was that nobody could see these packets and diagnose the problem correctly. If you had temporarily allowed-all into your security group and done a tcpdump you'd have gone "whoa!" and headed into the correct direction to fix the problem. Interesting...
Post reply on HN