Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 254 posts

Re: AWS us-east-2 outage

#121

Post by water-your-self »

deathanatos wrote: post #52
jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

The AWS status board, posted elsewhere in the comments, seems to think this is an AZ outage, not a regional one. Edit: although, one of our vendors that uses AWS has said that they think ELB registration is impacted (but I don't recall if that's regional?) and R53 is impacted (which is supposed to be global, IIRC). Dunno how much truth there is to it as we don't use AWS directly.

AWS is notorious did underreporting and failing to report. They do not have asafe culture and its bad for your career if there is a major outage

Re: AWS us-east-2 outage

#122

Post by mrits »

water-your-self wrote: post #115
dangero wrote: post #101

Earlier quoted context omitted.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

It gives me a bad gut feeling when you imply that multiple instances of a service is more complex than a single instance which cannot be duplicated easily. I also disagree that it is inherently more costly to run a service in multiple locations.

You should get into the database business. A lot of money to be made there if things are so trivial for you.

Re: AWS us-east-2 outage

#123

Post by wkoszek »

water-your-self wrote: post #115
dangero wrote: post #101

Earlier quoted context omitted.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

It gives me a bad gut feeling when you imply that multiple instances of a service is more complex than a single instance which cannot be duplicated easily. I also disagree that it is inherently more costly to run a service in multiple locations.

How do you NOT pay more for running double of everything + load balancers?

Re: AWS us-east-2 outage

#124

Post by sosodev »

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

I don't think there's a shortage of people who can architect reliable services. I think companies simply put reliability on the back burner because it rarely bites them. It's the same reason technical debt is so rarely paid off.

Re: AWS us-east-2 outage

#125

Post by jffry »

EddySchauHai wrote: post #75
Syonyk wrote: post #40

Earlier quoted context omitted.

And the reason that works is because HN is mostly hosted on its own stuff, without weird dependencies on anything beyond "the servers being up" and "TCP mostly working."

I believe it's on AWS after its two servers broke at the same time the other day.

Yep

  $ host news.ycombinator.com
  news.ycombinator.com has address 50.112.136.166
  
  $ host 50.112.136.166
  166.136.112.50.in-addr.arpa domain name pointer ec2-50-112-136-166.us-west-2.compute.amazonaws.com.

Re: AWS us-east-2 outage

#127

Post by nijave »

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

As some others have alluded to, it seems common AWS services (the ones you rely on to manage multi-AZ traffic like ALBs and Route53) spike in error rate and nose dive in response time so it becomes difficult to fail things over. On top of that, services like RDS that run active hot standby then rely on those to fail over so it's difficult to get the DB to actually fail over.

I suspect, behind the scenes, AWS fails to absorb the massive influx in requests and network traffic as AZs shift around.

I would think regions with more AZs (like us-east-1) would handle an AZ failure better since there's more AZs to spread the load across

What's more surprising, imo, is the large apps like New Relic and Zoom that you'd expect to be resilient (multi region/cloud) taking a hit

Re: AWS us-east-2 outage

#128

Post by _joel »

water-your-self wrote: post #115
dangero wrote: post #101

Earlier quoted context omitted.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

It gives me a bad gut feeling when you imply that multiple instances of a service is more complex than a single instance which cannot be duplicated easily. I also disagree that it is inherently more costly to run a service in multiple locations.

Of course it's more costly, you need to ensure state between locations so by virtue there's more infra to pay for.

It's not just a single instance too, there's generally a lot more infrastructure (db servers, app servers, logging and monitoring backends, message queues, auth servers... etc)

Re: AWS us-east-2 outage

#129

Post by mnordhoff »

alex_young wrote: post #99
acwan93 wrote: post #65

Earlier quoted context omitted.

In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?

Looks like this particular issue was due to power loss, and for power us-west-2 has one clear advantage: It's power is directly from the Columbia river and highly unlikely to have demand based outages.

Maybe not the entire region. Amazon was reportedly building a data center complex next to the natural gas Hermiston Generating Plant some distance from the river.

Re: AWS us-east-2 outage

#130

Post by sergiotapia »

fishnchips wrote: post #55
jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Stay in us-east-1, they provide Chaos Monkey for free. It's a feature.

I'm moving to us-weast-1
Post reply on HN