Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

Post reply on HN
Print view View original

171–180 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#171

Post by jey »

nasmorn wrote: post #163
jey wrote: post #51

Earlier quoted context omitted.

Yeah, it's got to be much larger than the other regions, so it makes sense that we see more errors. Since error_rate = machines * error_rate_per_machine.

The whole region is down, you just calculated the chance of at least one machine having an error.

No, I calculated the error rate for the region. If us-east-1 has 5 times the machines (or availability zones, or routers, or EBS backplanes, or other thing-that-can-fail) as us-west-1, we would expect to see us-east-1 have each type of error occur about 5 times as often as us-west-1.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#172

Post by override »

jmvoodoo wrote: post #29

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…

It's an EBS issue and something with ELB's as well.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#173

Post by rynop »

modsearch wrote: post #88
jmvoodoo wrote: post #29

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…

Our multi-AZ RDS instance did not failover correctly this time although it has in the past..

same here, no fail-over

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#175

Post by Terretta »

jamesaguilar wrote: post #132
calinet6 wrote: post #116

Earlier quoted context omitted.

If it's systematically difficult to do it correctly, then the system is wrong.

. . . or the problem is inherently complex.

> . . . or the problem is inherently complex.

You're close. Put another way, "inherent complexity is the problem."

What I mean by that is, the more your system is coupled, the more it is brittle.

Frankly, this is AWS's issue. It is too coupled: RDS relies on EBS, the console relies on both, etc. Any connection between two systems is a POF and must be architected to let those systems operate w/o that connection. This is why SMTP works the way it does. Real time service delivery isn't the problem, but counting on it is.

Uncouple all the things!

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#177

Post by jamesaguilar »

Terretta wrote: post #175
jamesaguilar wrote: post #132

Earlier quoted context omitted.

. . . or the problem is inherently complex.

> . . . or the problem is inherently complex. You're close. Put another way, "inherent complexity is the problem." What I mean by that is, the more your system is coupled, the more it is brittle. Frankly, this is AWS's issue. It is too coupled: RDS relies on EBS, the console relies on both, etc. Any connection between two systems is a POF and must be architected to let those systems operate w/o that connection. This…

Depends. Generic interfaces and non-reliance have costs too. In general I agree that things should be decoupled, but it's not always easy or practical.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#179

Post by j45 »

caseysoftware wrote: post #46
benwerd wrote: post #30

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

We (Twilio) have released a number of articles & presentations in this area: http://www.slideshare.net/twilio/highavailability-infrastruc... http://www.twilio.com/engineering/2011/04/22/why-twilio-wasn... It's strategy as opposed to how-to but the principles apply.

Sweet, thanks for sharing!
Post reply on HN