This feels the same way as hearing that the whole Internet just got shut down.
Amazon Web Services are down
11–20 of 346 posts
Re: Amazon Web Services are down
#12Yes, they are. :(
Re: Amazon Web Services are down
#13Earlier quoted context omitted.
That seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.
AWS randomize the zones per account, so "your" -1b is not necessarily the same as "my" -1b. I'm only seeing problems in my -1c. Are you seeing 3 zones failing all under the same account? If multiple AZs are down, AWS are going to have some serious explaining to do...
We (reddit) are seeing failures in all zones.
Re: Amazon Web Services are down
#14I guess this is one Reddit outage that can't be blamed on poor scaling
Looks like troubles in only one availability zone.
AWS have now confirmed that this affects multiple availability zones. From the status page: "..impacting EBS volumes in multiple availability zones in the US-EAST-1 region"
Re: Amazon Web Services are down
#15I guess this is one Reddit outage that can't be blamed on poor scaling
Thankfully, no. :)
But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.
Re: Amazon Web Services are down
#16Yay for relying on the cloud \o/
Re: Amazon Web Services are down
#17I guess this is one Reddit outage that can't be blamed on poor scaling
Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime. It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.
EBS seems to be the main problem here, I'll cite a former reddit employe (first comment on the blog that talked about EBS problems).
I don't work for reddit anymore (as of about a week ago, although I didn't get as much fanfare as raldi did), but I can tell you that they're giving Amazon too much credit here. Amazon's EBSs are a barrel of laughs in terms of performance and reliability and are a constant (and the single largest) source of failure across reddit. reddit's been in talks with Amazon all the way up to CIOs about ways to fix them for nearly a year and they've constantly been making promises that they haven't been keeping, passing us to new people (that "will finally be able to fix it"), and variously otherwise been desperately trying to keep reddit while not actually earning it.
Source: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...
cache: - scroll down for comment - http://webcache.googleusercontent.com/search?q=cache:cfbs-sp...
Re: Amazon Web Services are down
#18I guess this is one Reddit outage that can't be blamed on poor scaling
How does Reddit display the 'offline' page if it's down?
Re: Amazon Web Services are down
#19Earlier quoted context omitted.
That seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.
AWS randomize the zones per account, so "your" -1b is not necessarily the same as "my" -1b. I'm only seeing problems in my -1c. Are you seeing 3 zones failing all under the same account? If multiple AZs are down, AWS are going to have some serious explaining to do...
"AWS randomize the zones per account"
Interesting. This is news to me.
I found some more info here (Google cache since alestic.com is presently unreachable): http://webcache.googleusercontent.com/search?q=cache:0jxzyFj...