AWS us-east-2 outage • Hacker News Board

Re: AWS us-east-2 outage

#101

Post by dangero » Thu, Jul 28, 2022, 6:40 PM UTC

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks.

Zone downtime still falls under an AWS SLA so you know about how much downtime to accept and for a lot of businesses that downtime is acceptable.

Re: AWS us-east-2 outage

#102

Post by vineyardmike » Thu, Jul 28, 2022, 6:42 PM UTC

yodsanklai wrote: post #93

jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Naive question: don't people who care about resiliency have their services in more than one datacenter? or datacenter failure is considered such a rare event that's it's not worth the cost/trouble of using more?

Some people care about it but not enough to justify the added downsides - multi-data center is expensive (you pay per data center) and it’s complex (data sharding/duplication/sync).

If you’re Amazon where every second is millions of $ in transactions you care more than StartUp that has 1 request per minute. Even if you accept the risk, you still care when your DC goes down.

Also, a large chunk of AWS is managed from a single data center so if that one goes down you may still have issues with your service in another data center.

Re: AWS us-east-2 outage

#103

Post by ajanuary » Thu, Jul 28, 2022, 6:42 PM UTC

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Not looked into it too closely yet, but for us it looks like there were also issues connecting between the two remaining AZ in our 3 node cluster.

Re: AWS us-east-2 outage

#104

Post by bearjaws » Thu, Jul 28, 2022, 6:43 PM UTC

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Will need to see the post-mortem, when us-east-1 had its last big outage multiple AZs were working, but cross AZ functionality (lambda, event bridge) were impacted... which made recovery problematic.

Re: AWS us-east-2 outage

#105

Post by rjh29 » Thu, Jul 28, 2022, 6:45 PM UTC

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

It's a game theory thing. If everyone stays single AZ, everyone goes down at the same time so nobody gets blamed. Somehow the blame falls on AWS instead!

Re: AWS us-east-2 outage

#106

Post by rjh29 » Thu, Jul 28, 2022, 6:46 PM UTC

arecurrence wrote: post #70

acwan93 wrote: post #65

Earlier quoted context omitted.

In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?

Classically, us-east-1 received most of the hate given its immense size (it used to be several times larger than any other) and status as the first large aws data center. It also seemed to launch new aws features first but that may have been my imagination. If true, I'm sure always running the latest builds was not great for stability. us-west-2 has had outages as well but it is less common, even rare. I've been push…

Maybe Amazon should make us-east-1's actual datacenter change depend on the customer, as they do with the AZs :P

Re: AWS us-east-2 outage

#107

Post by latchkey » Thu, Jul 28, 2022, 6:46 PM UTC

mhb wrote: post #98

latchkey wrote: post #77

Earlier quoted context omitted.

Our best was a bird landing on a transformer up on a pole. Installed a fake Eagle after that.

Given the scope of the effort invested in attempting to prevent duck and goose crap on the world's docks, I'm skeptical that this tactic is effective.

Shrug... the datacenter is land locked (different animal species) and the problem hasn't happened again in multiple years.

I think you're taking the Eagle a bit too seriously though... if we didn't do anything how would we know? It isn't like this was an expensive thing to try out.

Re: AWS us-east-2 outage

#108

Post by jedberg » Thu, Jul 28, 2022, 6:47 PM UTC

yodsanklai wrote: post #93

jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Naive question: don't people who care about resiliency have their services in more than one datacenter? or datacenter failure is considered such a rare event that's it's not worth the cost/trouble of using more?

AWS makes it pretty easy to operate in multiple AZs within a region (each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other).

That being said, there is still an added cost and complexity to operate in multiple AZs, because you have to synchronize data across the AZs. Also you have to have enough reserved instances to move into when you lose an AZ, because if you're running lean and each zone is serving 33% of your traffic, suddenly the two that are left need to serve 50% each.

The bigger companies with overhead reservations will get all the instances before you can launch any on demand during an AZ failure.

Re: AWS us-east-2 outage

#109

Post by [deleted] » Thu, Jul 28, 2022, 6:47 PM UTC

yodsanklai wrote: post #93

jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Naive question: don't people who care about resiliency have their services in more than one datacenter? or datacenter failure is considered such a rare event that's it's not worth the cost/trouble of using more?

[deleted]

Re: AWS us-east-2 outage

#110

Post by ilikecakeandpie » Thu, Jul 28, 2022, 6:47 PM UTC

dangero wrote: post #101

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

This is true, but I think it would be more acceptable if the region were down vs the single AZ