Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 254 posts

Re: AWS us-east-2 outage

#211

Post by oxfordmale »

arecurrence wrote: post #70
acwan93 wrote: post #65

Earlier quoted context omitted.

In all seriousness, we've been deploying everything on us-west-2, and it seems to have dodged most of the outages recently. Is there something special about that data center?

Classically, us-east-1 received most of the hate given its immense size (it used to be several times larger than any other) and status as the first large aws data center. It also seemed to launch new aws features first but that may have been my imagination. If true, I'm sure always running the latest builds was not great for stability. us-west-2 has had outages as well but it is less common, even rare. I've been push…

I believe us-east-1 runs some of the control plane and an us-east outage can effectively take a service in a different region offline as it can break IAM authentication

Re: AWS us-east-2 outage

#212

Post by throwbigdata »

aledalgrande wrote: post #169
_joel wrote: post #128

Earlier quoted context omitted.

Of course it's more costly, you need to ensure state between locations so by virtue there's more infra to pay for. It's not just a single instance too, there's generally a lot more infrastructure (db servers, app servers, logging and monitoring backends, message queues, auth servers... etc)

also inter region replication costs bandwidth money

Lots and lots of money.

Re: AWS us-east-2 outage

#213

Post by nodesocket »

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

It's always more complicated than just deploying EC2 instances into multiple-az's. Here are some things I noticed from today's events.

First: RDS. I saw one of our RDS instances do a failover to the secondary zone because the primary was in the zone that had the power outage. RDS failovers are not free and have a small window of downtime (60-120s as claimed by AWS[1]).

Second: EKS (Kubernetes). One of our Kubernetes EC2 worker nodes (in EKS) went down because it was in the zone with the power outage. Kubernetes did a decent job at re-scheduling pods, but there were edge cases for sure. Mainly with Consul and Traefik running inside of the Kubernetes cluster. Finally, when the Kubernetes EC2 worker node came back up, nearly nothing got scheduled back to it. I had to manually re-deploy to get pod distribution even again. Though the last issue might be something I can improve on by using the new Kubernetes attribute topologySpreadConstraints[2].

[1] https://aws.amazon.com/premiumsupport/knowledge-center/rds-f... [2] https://kubernetes.io/docs/concepts/scheduling-eviction/topo...

Re: AWS us-east-2 outage

#214

Post by throwbigdata »

gamblor956 wrote: post #199
Nextgrid wrote: post #160

Earlier quoted context omitted.

I have 2 takes on this: 1) AWS is already really expensive, just on a single AZ. Replicating to a second AZ would almost double your costs. I can't help but bring up the point that an old-school bare-metal setup on something like Hetzner/OVH/etc becomes significantly more cost-effective since you're not using AWS's advantages in this area anyway (and as we've seen in practice, AWS is nowhere near more reliable - how…

Going bare-metal is a premature optimization. Most startups that go that route don't survive long enough to make use of this optimization. Take advantage of AWS (or Azure, or DO) until you're big enough that bringing the action in-house is a financially and technically prudent option.

It’s premature when it’s premature. It’s late when it’s not.

Re: AWS us-east-2 outage

#215

Post by martinald »

macintux wrote: post #187
outworlder wrote: post #176

Earlier quoted context omitted.

> The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. Not really. What's more likely is that their companies have other priorities. Multi-AZ architectures are more expensive to run, but that's normally not the issue. What's really costly is testing their assumptions. Sure, by…

Or, the redundancy actually causes a failure, so not only have you spent more money but you’ve reduced your availability doing so. (Or worse, the redundancy causes a subtle failure like data loss.)

Nail on the head. The amount of times I've seen way overcomplicated redundancy setups which fail in weird and wonderful ways, causing way more downtime than just a simplier setup is pretty silly.

Re: AWS us-east-2 outage

#216

Post by aledalgrande »

xchaotic wrote: post #206
aledalgrande wrote: post #185

Earlier quoted context omitted.

Yeah the data stores are the ones that I would always keep multi AZ no matter what. Everything else is stateless and can be moved quickly.

Write an article on that because you make it sound simple. Or better yet, start a company that configures this for companies.

Nothing is inherently simple.

Depending on the size of the company it can be simple or hard. Most companies that need this are not huge. Things like RDS, Elasticache, ECR and Secrets have multi AZ integrated so not hard to do it. If you operate on ECS or EKS it's pretty straightforward to boot up nodes and load balancers in another AZ.

Maybe you have a system that requires more hands on work and want to explain your point of view? I don't appreciate the snarky responses tho.

Re: AWS us-east-2 outage

#217

Post by Leires »

poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

I can tell you from experience that the cloud architects are world class and it's actually the data techs that are the problem. Amazon doesn't value data center techs, they don't pay competitively and hire techs that barely have enough skill so they can pay them nothing. Then they metric the fuck out of the teams so that everyone focuses on quick fixes instead of taking the time to troubleshoot long-term persistent issues. Couple this with the fact that management is only concerned with creating new capacity instead of fixing existing capacity.

Re: AWS us-east-2 outage

#218

Post by throwbigdata »

harryh wrote: post #139
rjh29 wrote: post #105

Earlier quoted context omitted.

It's a game theory thing. If everyone stays single AZ, everyone goes down at the same time so nobody gets blamed. Somehow the blame falls on AWS instead!

I think you're confusing availability zones with regions in this comment. AWS AZs don't even have consistent naming across AWS accounts.

That’s a feature, not a bug. If we all had the same number one ; then things would not be loaded anything close to evenly. There is some command to find out what the unique ID number is for your particular zones with your naming.

Re: AWS us-east-2 outage

#219

Post by smt88 »

gtirloni wrote: post #161
dmalvarado wrote: post #126

I dunno how else to put it. Having EVERYTHING on AWS is a national security threat. This isn't good, and someone who can do something about it needs to.

Good thing we don't have EVERYTHING on AWS, so no threat detected.

Our own applications are hosted on Azure, but we had an outage today anyway. It was because apparently Netlify and Auth0 use AWS and went down, which took down our static sites and our authentication.

The nature of our business means it wasn't a big deal, but I could imagine lots of people were in the same boat.

Re: AWS us-east-2 outage

#220

Post by bad416f1f5a2 »

martinald wrote: post #215
macintux wrote: post #187

Earlier quoted context omitted.

Or, the redundancy actually causes a failure, so not only have you spent more money but you’ve reduced your availability doing so. (Or worse, the redundancy causes a subtle failure like data loss.)

Nail on the head. The amount of times I've seen way overcomplicated redundancy setups which fail in weird and wonderful ways, causing way more downtime than just a simplier setup is pretty silly.

Don’t make the mistake of overromanticizing the simple solutions. They have nice, well understood failure conditions, and they come up relatively frequently.

When you start playing the HA game, the easy failures go off the table, and things break less often because “failures happen constantly and are auto-healed”. But when your virtual IP failover goes sideways or your cluster scheduler starts reaping systems because the metadata service is giving it useless data, you’re well into an infrequent, complex failure, and I hope you have a good ops team.

It’s always a trade off.

Post reply on HN