Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

81–90 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#81
post #4

Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.

I would agree with you, but Amazon is just downright dishonest in their reports, which makes me sad, because I love Amazon. Go look at the past reports, they've never shown a red market, only "degraded performance" even when services for multiple availability zones went down at the same time due to their power outage (so had you architected to multiple AZs you were still fucked). When they have a single AZ go down, they won't even give it a yellow marker on the status page, they'll just put a footnote on a green marker. It makes their status dashboard pretty much useless for at a glance checking (why even have colors if they don't mean anything?)

Read their report from the major outage earlier this year, they start out by saying "elevated error rates", when many services were in fact down, and it wasn't until hours later they finally admitted to having an issue that affected more than just one availability zone.

From Forbes: ”We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone.” By 11:49 EST, it reported that, ”Power has been restored to the impacted Availability Zone and we are working to bring impacted instances and volumes back online.” But by 12:20 EST the outage continued, “We are continuing to work to bring the instances and volumes back online. In addition, EC2 and EBS APIs are currently experiencing elevated error rates.” At 12:54 AM EST, AWS reported that “EC2 and EBS APIs are once again operating normally. We are continuing to recover impacted instances and volumes.”

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#83
post #74
post #4

Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.

I'm also having API and Console timeouts, so I would disagree that it's limited to a single AZ.

The API and Console timeouts are likely due to high load from everyone logging in to see what's going on.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#84
This is affecting more than a single Availability Zone, but probably for reasons that have been seen before. One reason might be that an EBS failure in one AZ triggers a spike in EBS activity in other AZs which overwhelms EBS. (I believe this is what happened in April 2011).

Does anybody have any experience with migrating to Oregon or N. California in terms of speed and latency?

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#85
As https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager.

Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud.

(And many other relevant quotes.)

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#88

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…

Our multi-AZ RDS instance did not failover correctly this time although it has in the past..

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#90
post #43

pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."

Tons has been written about Netflix's architecture, and they're down as well. I believe Amazon isn't being totally transparent.
Post reply on HN