Amazon Web Services are down

#1

Post by yuvadam » Thu, Apr 21, 2011, 8:51 AM UTC

Amazon Web Services are down
status.aws.amazon.com

Re: Amazon Web Services are down

#2

Post by yuvadam » Thu, Apr 21, 2011, 8:52 AM UTC

Current status: bad things are happening in the North Virginia datacenter.

EC2, EBS and RDS are all down on US-east-1.

Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

Re: Amazon Web Services are down

#3

Post by powdahound » Thu, Apr 21, 2011, 9:14 AM UTC

I'm seeing 1 EBS server out of 9 having issues (5 in one availability zone, 4 in another). CPU wait time on the instance is stuck at 100% on all cores since the disk isn't responding. Sounds like others are having much more trouble.

Re: Amazon Web Services are down

#4

Post by vnorby » Thu, Apr 21, 2011, 9:33 AM UTC

From EngineYard: "It looks like EBS IO in the us-east-1 region is not working ideally at this point. That means all /data and /db Volumes which use EBS have bad IO performance, which can cause your sites to go down."

Re: Amazon Web Services are down

#5

Post by kylec » Thu, Apr 21, 2011, 9:34 AM UTC

I guess this is one Reddit outage that can't be blamed on poor scaling

Re: Amazon Web Services are down

#6

Post by thezilch » Thu, Apr 21, 2011, 9:39 AM UTC

kylec wrote: post #5

I guess this is one Reddit outage that can't be blamed on poor scaling

Looks like troubles in only one availability zone.

Re: Amazon Web Services are down

#7

Post by mechanical_fish » Thu, Apr 21, 2011, 9:45 AM UTC

thezilch wrote: post #6

kylec wrote: post #5

I guess this is one Reddit outage that can't be blamed on poor scaling

Looks like troubles in only one availability zone.

That seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.

Re: Amazon Web Services are down

#8

Post by StavrosK » Thu, Apr 21, 2011, 9:53 AM UTC

kylec wrote: post #5

I guess this is one Reddit outage that can't be blamed on poor scaling

Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime.

It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.

Re: Amazon Web Services are down

#9

Post by justinsb » Thu, Apr 21, 2011, 9:55 AM UTC

mechanical_fish wrote: post #7

thezilch wrote: post #6

Earlier quoted context omitted.

Looks like troubles in only one availability zone.

That seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.

AWS randomize the zones per account, so "your" -1b is not necessarily the same as "my" -1b. I'm only seeing problems in my -1c. Are you seeing 3 zones failing all under the same account?

If multiple AZs are down, AWS are going to have some serious explaining to do...

Re: Amazon Web Services are down

#10

Post by muppetman » Thu, Apr 21, 2011, 10:00 AM UTC

StavrosK wrote: post #8

kylec wrote: post #5

I guess this is one Reddit outage that can't be blamed on poor scaling

Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime. It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.

Slightly offtopic, but wasn't that post by an ex-employee as a comment? Not that the technical aspect of it wasn't fantastic, because it was, but I don't think Reddit said anything publically did they?