Two of our four EC2 servers are down. Both have RDS connections and EBS storage. Looks like the RDS connection has something to do with it...
RDS is built on EBS, so if EBS has problems you can bet that RDS will feel the effects.
Amazon EC2 currently down. Affecting Heroku, Reddit, Others
111–120 of 311 posts
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#112Earlier quoted context omitted.
And Netflix's Simian Army is awesome: http://techblog.netflix.com/2011/07/netflix-simian-army.html They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-in...
Yet netflix is currently down, so maybe there is a problem with the chimp army?
The problem is likely the same as usual: if the damn control plane is down, it doesn't matter how robust your failover architecture is, because your requests to bring up new machines go unanswered.
There's pretty much no way to architect around that one as an AWS user (apart from going fully multi-cloud, but "nobody" actually does that, at least at scale), and I'm kind of shocked that those bits of AWS are still not robust against "single AZ outages", given that they're involved in pretty much every one of these incidents and make them affect people on the entire cloud...
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#113This reminds me of an ad campaign we sometimes run on reddit:
http://www.reddit.com/comments/hg9oa/your_platform_is_on_aws...
... which you'll be able to click as soon as reddit is back up :)
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#114The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.
I'm surprised amazon hasn't built another region in the east. If you're in the west you get US-West-1 and US-West-2 and can failover and distribute between the two, why don't they have that kind of duplication in the east?
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#115Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#116Earlier quoted context omitted.
One thing we discovered this morning: it appears the AWS console itself is hosted in N Virginia. This means that if you were trying to make changes to your EC2 instances in the West using the GUI, you couldn't, even though the instances themselves were unaffected.
shouldn't amazon themselves have architected their own app to be able to move around? I get tired of the snipes from people that "well, you're doing it wrong", as if this is trivial stuff. But if Amazon themselves aren't even making their AWS console redundant between locations, how easy/straightforward is it for anyone else? To what extent is this just "the cobbler's kids have no shoes?"
If it's systematically difficult to do it correctly, then the system is wrong.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#117Unfortunate.
I have to deal with a number of folks who will be overjoyed to read this news when their tech cartel vendor of choice forwards it this evening.
There's a huge contingent of currently endangered infrastructure folks (and vendors who feed off them) out there who throw a party every time AWS has a visible outage.
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#118Heh, the cynical side of me would like to point out that this is a great way to get people to stop talking about wiping a user's kindle. :-)
Because reddit and ycombinator are the center of the tech sociosphere?
Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others
#119One of the most frustrating issues here is that we have to deal with Amazon's status page for information. It's a complex page, divided by continent instead of region, which means at least 5 or 6 clicks to figure out progress. They should learn from these issues about how people want to be informed - to date, they haven't. Also, they have a twitter account, which would be the perfect fit to keep everyone up-to-date with what's going on; to at least show a human side to these issues. Alas, they're not updating that either.
I've been working with AWS since early 2006 when they first launched - I was lucky to be granted a VIP invite to try out EC2 before everyone else, and ended up launching the first public app on EC2. This might be the first time when frustration has overcome my love for these guys.