Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS EC2/RDS Outage in us-east-1

status.aws.amazon.com

Post reply on HN
Print view View original

81–90 of 152 posts

Re: AWS EC2/RDS Outage in us-east-1

#81

Post by acdha »

matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

This should always be a business calculation but I do think you should note that there’s at least an order of magnitude difficulty increase between multiple AZs and regions, especially if you’re using services like RDS where it’s designed in, so I’d consider that a solid bridge step.

I’m trying to put some numbers into that, I’ve been running a relatively well trafficked website in multiple AZs since 2011. We had ~20 minutes of downtime when they had a network routing issue for us-east-1 and a few hours of degraded service when S3 had a region-wide outage. I haven’t added up the number of single AZ outages during that period but based on the RSS feeds I think it’s a good bit more relative to the very modest additional cost.

Re: AWS EC2/RDS Outage in us-east-1

#83

Post by me_pie_three »

bkovacev wrote: post #53
btown wrote: post #4

https://status.heroku.com/incidents/1892 - it appears Heroku is being particularly affected. We've had multiple sites on multiple accounts go down in the past few minutes. EDIT T16:31Z: It appears Heroku has failed over their dashboard, but dynos are still failing to come online. We had assumed that they had multi-region failovers for their customers. Incredibly disappointing.

We cannot even restart/turn off the dynos or get into the dashboard to turn off and kill our background tasks for some of our clients.

Heroku: don't restart your app; restarting is broken

Also Heroku: we auto restart your dyno every day!

So now my app is down because Heroku forced it to restart, and none of our hourly employees can work :|

Re: AWS EC2/RDS Outage in us-east-1

#87

Post by Dunedan »

karmakaze wrote: post #84

I remember reading about how not all AWS regions are similarly operated and that one was a snowflake. Is it US-East-1?

Yes. us-east-1 is the first AWS region Amazon made publicly available. It's also historically used by a lot of customers as "default region" where they launch all workloads where they don't have special needs of launching them somewhere else.

That has lead to us-east-1 being the largest AWS region by far, also compromised of the largest number of availability zones (6) of all AWS regions.

Re: AWS EC2/RDS Outage in us-east-1

#89

Post by ihaveajob »

holykin wrote: post #21
rvz wrote: post #16

Well, this outage says something about the companies that religiously depend on it. If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

Multi AZ would have also sufficed. This appears localized to one zone, not the whole region.

Several people report problems with multizone deployments, so it seems like AWS is downplaying this.

Re: AWS EC2/RDS Outage in us-east-1

#90

Post by freehunter »

manojlds wrote: post #75
matt2000 wrote: post #61

Just wanted to add a quick note before we get the usual deluge of "you should be running in multiple AZs and regions" posts: These outages are relatively rare and your best decision might just be to accept the tiny amount of downtime and keep your app simple and inexpensive to run. I of course don't know the tradeoffs involved in running your system, but I know for a lot of my situations the simplicity of single AZ w…

Being on AWS is also easy to explain to customers about downtimes - AWS was down and customers are pretty understanding in that case and don't demand why you aren't multi AZ etc ( of course YMMV based on sensitivity of your business)

Kind of like the "no one ever got fired for recommending IBM", if you have significant downtime on Linode or Hetzner, people are going to ask "why weren't you on AWS?!". If you're on AWS and AWS goes down, you get to skip that question entirely. You were already using the logical choice, you don't need to defend anything.

If Netflix can go down when AWS goes down, so can your app. AWS outages impact so much of the Internet that people will just accept it.

Of course here I am with a site on Heroku which uses AWS... impacted by the AWS outage... fielding questions about why I didn't pick AWS if Heroku suffers outages like this. Can't please them all.

Post reply on HN