Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 254 posts

Re: AWS us-east-2 outage

#111

Post by jozzy-james »

ajanuary wrote: post #103
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Not looked into it too closely yet, but for us it looks like there were also issues connecting between the two remaining AZ in our 3 node cluster.

we definitely had issues with all of the AZs in east-2, and far more services impacted than just EC2 (f.e. RDS, elasticache were intermittently down for us)

Re: AWS us-east-2 outage

#112

Post by papito »

jedberg wrote: post #44

Sorry all I jinxed it. Yesterday I was in a meeting and said "The only regional outages AWS has ever had were in us-east-1, so we should just move to us-east-2." Now I guess we have to move to us-west-2. :) Update: looks like it's only one zone anyway, so my statement still stands!

Please don't move to us-west. We are probably going to have an 11-point earthquake the next day.

Thanks!

Re: AWS us-east-2 outage

#113

Post by bilalq »

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

It's not so cut-and-dried. The AZ isolation guarantees are not quite at the maturity they need to be.

If you're using any managed services by AWS, you need to rely on their own services to be AZ fault-tolerant. In AWS speak, they may well be (just with elevated error rates for a few minutes while load balancing shifts traffic away from a bad AZ). But as an AWS customer, you still feel the impact. As an example, one of our CodePipelines failed the deployment step with an InternalError from CloudFormation. However, the actual underlying stack deployment succeeded. When we went to retry that stage, it wouldn't succeed because the changeset to apply is no more. It required pushing a dummy change to unblock that pipeline.

Similarly, many customers run Lambdas outside of VPCs that theoretically shouldn't be tied to an AZ. You're still reliant on the AWS Lambda team to shift traffic away from a failing AZ, and until they do that, you'll see "elevated error rates" as well.

Re: AWS us-east-2 outage

#114

Post by poxrud »

jozzy-james wrote: post #111
ajanuary wrote: post #103

Earlier quoted context omitted.

Not looked into it too closely yet, but for us it looks like there were also issues connecting between the two remaining AZ in our 3 node cluster.

we definitely had issues with all of the AZs in east-2, and far more services impacted than just EC2 (f.e. RDS, elasticache were intermittently down for us)

Both RDS and elasticache run on EC2. But both of them have Multi-AZ options.

Re: AWS us-east-2 outage

#115

Post by water-your-self »

dangero wrote: post #101
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

For most businesses a little down time here and there is a calculated risk versus more complex infrastructure. You can’t assume all the cloud architects are idiots — they have to report their task list and cost of infrastructure to someone who can give feedback on various options based on comparative resource requirements and risks. Zone downtime still falls under an AWS SLA so you know about how much downtime to acc…

It gives me a bad gut feeling when you imply that multiple instances of a service is more complex than a single instance which cannot be duplicated easily.

I also disagree that it is inherently more costly to run a service in multiple locations.

Re: AWS us-east-2 outage

#116

Post by jonatron »

I just set up a few small sites (not live yet) on us-east-2, because us-east-1 has a poor reputation. I wanted to avoid multi-region to keep things simple, but now I'm thinking I might have to spend the additional time on it. Not ideal when there's no dedicated ops.

Re: AWS us-east-2 outage

#117

Post by water-your-self »

rjh29 wrote: post #105
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

It's a game theory thing. If everyone stays single AZ, everyone goes down at the same time so nobody gets blamed. Somehow the blame falls on AWS instead!

Best take lol

Re: AWS us-east-2 outage

#118

Post by eurasiantiger »

datatrashfire wrote: post #84
eurasiantiger wrote: post #62

Earlier quoted context omitted.

Thanks, this comment made it very clear to me that I never want to touch a terraform system.

I think any system is susceptible to problems like this if the underlying hardware becomes unavailable. Using dynambodb to obtain locks on s3 is a pretty common pattern in AWS development. This has more to do with AWS than Terraform.

Rarely is Terraform mentioned in any other context.

Re: AWS us-east-2 outage

#119

Post by easton »

jasonjayr wrote: post #61
eatonphil wrote: post #30

Wonder if this is why Zoom is down. Wasn't able to connect just now. The connection proxy/sites were giving 504s.

IIRC Zoom signed up with Oracle Cloud when COVID hit and they needed to scale like crazy. https://www.oracle.com/customers/zoom/ I'm not sure if Zoom has any Critical infra in AWS though.

I interviewed there a few months ago for DevOps, and one of the people I interviewed with said that most of Zoom was in AWS (they liked that I had AWS stuff on my resume).

Re: AWS us-east-2 outage

#120

Post by Tehnix »

jedberg wrote: post #108
yodsanklai wrote: post #93

Earlier quoted context omitted.

Naive question: don't people who care about resiliency have their services in more than one datacenter? or datacenter failure is considered such a rare event that's it's not worth the cost/trouble of using more?

AWS makes it pretty easy to operate in multiple AZs within a region (each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other). That being said, there is still an added cost and complexity to operate in multiple AZs, because you have to synchronize data across the AZs. Also you have to have enough reserved instances to move into when you lose an…

> each AZ is considered a separate datacenter but in real life each AZ is multiple datacenters that are really close to each other

For AWS specifically, I’m fairly certain they maintain a minimum distance and are much more strict on requirements to be on different grids etc than other Cloud providers. A few years ago they were calling out Azure and Google Cloud on exactly what you describe (having data centers essentially on the same street almost).

Post reply on HN