Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

Post reply on HN
Print view View original

51–60 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#51

Post by jey »

acangiano wrote: post #40
diego wrote: post #31

The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.

Realistically, it's at least in part because everyone defaults to the East region. So it's the most crowded and demanding on the system.

Yeah, it's got to be much larger than the other regions, so it makes sense that we see more errors. Since error_rate = machines * error_rate_per_machine.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#55

Post by nostromo »

diego wrote: post #31

The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.

One thing we discovered this morning: it appears the AWS console itself is hosted in N Virginia.

This means that if you were trying to make changes to your EC2 instances in the West using the GUI, you couldn't, even though the instances themselves were unaffected.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#57

Post by caseysoftware »

caseysoftware wrote: post #46
benwerd wrote: post #30

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.

And Netflix's Simian Army is awesome:

http://techblog.netflix.com/2011/07/netflix-simian-army.html

They've even released the "Chaos Monkey" open source: http://techblog.netflix.com/2012/07/chaos-monkey-released-in...

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#58

Post by onenine »

thomaslutz wrote: post #19
orourkek wrote: post #11

Earlier quoted context omitted.

Even if they do understate the issue, It's limited to US-EAST 1, and is an EBS issue. Saying that EC2 is "down" because of this is totally off the mark - I've got dozens of EBS volmes in EAST 1 that are unaffected, plus all of the other zones that are operating normally...

EC2 instances are affected by the EBS issues. But you're right, it's not correct that EC2 is "down".

Not all instances are EBS backed.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#59

Post by dsl »

res0nat0r wrote: post #21
driverdan wrote: post #8

Earlier quoted context omitted.

Amazon is downplaying the problem. It is affecting many large sites.

Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.

I'm seeing issues across multiple AZ's. Everything is still up, but that doesn't stop me from getting paged.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#60

Post by ejdyksen »

rynop wrote: post #41
gtaylor wrote: post #38

Does anyone know if this really is just one AZ? Seems like an awful lot of larger sites are down. I'd expect at least some of them to be multi-AZ by now.

I'm multi-az and am being impacted. I also use multi-az RDS adn its being impacted. So calling BS on the 1 az impact.

One of our Multi-AZ RDS instances failed over successfully. It was down for about 5-10 minutes.
Post reply on HN