Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

Post reply on HN
Print view View original

101–110 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#101

Post by obeattie »

acangiano wrote: post #40
diego wrote: post #31

The N. Virginia datacenter has been historically unreliable. I moved my personal projects to the West Coast (Oregon and N. California) and I have seen no significant issues in the past year. N. Virginia is both cheaper and closer to the center of mass of the developed world. I'm surprised Amazon hasn't managed to make it more reliable.

Realistically, it's at least in part because everyone defaults to the East region. So it's the most crowded and demanding on the system.

Yep, according to the most recent estimate I saw[1], us-east was more than twice the size of all other regions combined.

[1] http://huanliu.wordpress.com/2012/03/13/amazon-data-center-s...

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#102

Post by randomdata »

Ramonaxvh wrote: post #44

This is the kind of stuff I think about when I hear people talk about the cloud and promise that downtime is a thing of the past. Cloud hosting is not drastically different from any other type of service and is still vulnerable to the same problems.

I always understood the cloud to mean a black box of sorts that automatically handles failover, among other things. The cloud being just a fuzzy representation of the infrastructure.

S3 probably fits the description of a cloud service. You send your data, and the service worries about making it redundant without your intervention. If data in NE USA is unavailable, the service will automatically serve you the data from somewhere else. You don't need to know how it works.

EC2 and some of these other building blocks, however, I would not consider to be cloud services. Merely tools for building out your own cloud services to other customers who then shouldn't have to think about failover and other such concerns.

If you know you are using a server that is physically located in a certain geographic location, it need not be represented by a cloud. It is a distinct point on the network.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#104

Post by abailin »

jmvoodoo wrote: post #29

All of our EC2 hosts appear to be functioning fine, but they can't connect to their RDS instance which renders our app useless. If you scroll down the page you'll also see that RDS instances are having connectivity issues. Not sure if it's related but for RDS users the impact is far worse. EDIT: We are also using multi-AZ RDS, so either Amazon's claims for multi-AZ are bs, or their claims that this is only impacting…

Same here, our servers are still up, but our RDS instance is down.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#105

Post by kalid »

nowarninglabel wrote: post #81
ojiikun wrote: post #4

Per the linked dashboard, some instances in a single AZ in a single Region are having storage issues. Calling EC2 "down" is a bit dramatic, provided AMZN are being sincere with their status reports. Any system that can competently fail over to another AZ will be unaffected.

I would agree with you, but Amazon is just downright dishonest in their reports, which makes me sad, because I love Amazon. Go look at the past reports, they've never shown a red market, only "degraded performance" even when services for multiple availability zones went down at the same time due to their power outage (so had you architected to multiple AZs you were still fucked). When they have a single AZ go down, t…

It's like grade inflation. You can never give out an F (Mr. Admissions officer, are you so bad at your job that you would admit such an unqualified student?), so a Gentleman's C is handed around. In Amazon's case, it's a gentleman's B+ (green, with an info icon).

A: fine A-: problems B+: servers are on fire

I really like Amazon as a company, use a lot of their services, but this is dishonest.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#106

Post by eloisant »

btilly wrote: post #85

As https://twitter.com/DEVOPS_BORAT says, At conference you can able tell cloud devops by they are always leave dinner for respond to pager. Also, What is happen in cloud is stay in cloud because nobody can able reproduce outside of cloud. (And many other relevant quotes.)

To be fair, AWS downtime always make the news because they affect a lot of majors websites, but that doesn't mean an average sysadmin (or devops, whatever) would do better in term of uptime with his own bay and his toys.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#107

Post by hkarthik »

benwerd wrote: post #30

This would be a great time to post a guide to architecting systems for failover using AWS. Anyone got a great guide?

I wish there was. Architecting a system for failover requires a mindset change, a culture change within your org, and the right technology that lets you build for it while not slowing yourself down too much. The last point being the hardest.

I would argue that none of the common full stack frameworks that startups use are fault tolerant enough for AWS. Most of them have multiple failure points that can quickly bring down entire apps.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#108

Post by w_t_payne »

gfodor wrote: post #80
Ramonaxvh wrote: post #44

This is the kind of stuff I think about when I hear people talk about the cloud and promise that downtime is a thing of the past. Cloud hosting is not drastically different from any other type of service and is still vulnerable to the same problems.

If you go through the pains of architecting your system to span multiple AZs, or you avoid using EBS, then you probably dodge most of the EC2 outages. (Remains to be seen if that is the case here.) That said, I don't think most people think using the cloud means that downtime is a thing of the past. I think the more attractive proposition is when hardware breaks, or meteors hit the datacenter, etc, it is their proble…

Yeah. Even with this, they still do better than I would. My record: misconfigured air-conditioning unit alarm leading to servers being baked at high temperature over a weekend, leading to much wailing and gnashing of teeth. I now know to be really careful to set up air conditioning units properly, but what other lessons am I still waiting to learn? The main lesson that I took from this is that I should stick to what I am good at: cutting code & chewing data. :-)

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#109

Post by grumps »

bdcravens wrote: post #90
lr wrote: post #43

pg, or someone else from HN... Could you please edit this title for accuracy? Maybe, "Poorly designed sites taken out because of problems in one Amazon availability zone."

Tons has been written about Netflix's architecture, and they're down as well. I believe Amazon isn't being totally transparent.

Personally, starting to believe the Amazon is quickly becoming public enemy #2.
Post reply on HN