Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

71–80 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#71

I suppose this explains why Airbnb is down now, as well: http://aws.amazon.com/solutions/case-studies/airbnb/ I was just in the middle of booking a stay in a Palo Alto startup embassy for this week, too!

They've been down for several hours actually. I tried to get to their site this morning to check out the jedburg talk, and got the "We're on the case" message.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#73
post #43

pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."

This issue is affecting both an EC2 Zone and Amazon's RDS servers, which are technically multi-zone. There are a ton of well-architected apps and sites that have been affected. Unconstructive...

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#74
post #4

Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.

I'm also having API and Console timeouts, so I would disagree that it's limited to a single AZ.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#76

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…

Our multi-zoned RDS instance was able to fail over to another zone with minimal downtime. It took about 2 minutes.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#77
post #50

Earlier quoted context omitted.

http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.

The last Netflix post mortem mentioned they had a bug in their configuration where they kept sending traffic to already down ELB instances, which was the cause of the last outage for them if I remember correctly.

AirBnB is also down.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#78
post #50

Earlier quoted context omitted.

Many large sites which aren't properly architecting their infrastructure to deal with the commodity nature of the cloud.

http://downrightnow.com/netflix If netflix is down, then it's something most companies who know how to design fail over can't cope with.

Not necessarily, it could be some element of the Netflix architecture that due to their size and/or design trade-offs has taken longer / is harder to eliminate than it would be for others.

Other services, like Twilio, have come through several of these major problems with US-EAST generally unscathed while Netflix has had issues repeatedly.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#79
post #76

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…

Our multi-zoned RDS instance was able to fail over to another zone with minimal downtime. It took about 2 minutes.

Lucky you? :)

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#80

This is the kind of stuff I think about when I hear people talk about the cloud and promise that downtime is a thing of the past. Cloud hosting is not drastically different from any other type of service and is still vulnerable to the same problems.

If you go through the pains of architecting your system to span multiple AZs, or you avoid using EBS, then you probably dodge most of the EC2 outages. (Remains to be seen if that is the case here.)

That said, I don't think most people think using the cloud means that downtime is a thing of the past. I think the more attractive proposition is when hardware breaks, or meteors hit the datacenter, etc, it is their problem, not yours. You still have to deal with software-level operations, but hardware-level operations is effectively outsourced. The question is if you think you can do a better job than Amazon -- some companies think they can, most startups know they can't.

Post reply on HN