Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 254 posts

Re: AWS us-east-2 outage

#241

Post by throwbigdata »

saimiam wrote: post #233
throwbigdata wrote: post #223

Earlier quoted context omitted.

Wait. Are you saying that while AWS maintains multiple AZs they can’t maintain reliability on the failover systems between them?

Did you, by chance, reply to the wrong comment? Don’t think I said anything about failovers etc. The point made to me was that a devops role can be made to eventually automate their own job away to an extent. To an investor, having a devops role on staff is acceptable. If you never had a devops role and used AWS managed services, you can’t automate that and trim costs. I.e., devops roles look like surplus in the syst…

I did reply to the wrong comment, apologies!

Re: AWS us-east-2 outage

#242

Post by boomer918 »

retinaros wrote: post #178
boomer918 wrote: post #132

Earlier quoted context omitted.

A multi-az deployment is a checkbox in most AWS services, e.g. ASGs, RDS, load balancers, etc. Someone didn't check that box because they didn't know about it, there isn't much complexity in it.

multi az brings multi complexity in terms of data duplication, consistency, if your app wasnt designed to handle those kind of scenarios and experience high users loads then you are in for a lot of problems. designing for those scenarios increase complexity; cost; architecture style and most of the time it will bring you in microservices territory where most of the companies lack experience and just are following bes…

RDS just has a button for multi-AZ primaries. No complexity or microservices.

Re: AWS us-east-2 outage

#243

Post by rjh29 »

harryh wrote: post #139
rjh29 wrote: post #105

Earlier quoted context omitted.

It's a game theory thing. If everyone stays single AZ, everyone goes down at the same time so nobody gets blamed. Somehow the blame falls on AWS instead!

I think you're confusing availability zones with regions in this comment. AWS AZs don't even have consistent naming across AWS accounts.

Clarification: 1/3 of sites will go down (those using the AZ that went offline), but my point is the same. Most companies aren't using multiple AZs, let alone multiple regions.

Re: AWS us-east-2 outage

#244

Post by tatersolid »

fragmede wrote: post #46
CoastalCoder wrote: post #37

Earlier quoted context omitted.

> Looks like Snap, Crackle and Pop are down as well. I don't work on cloud stuff, so I'm genuinely unsure if this is a joke.

It's a joke but I only knew that because Snap is/was (as of S1) hosted on GCP and not AWS. Crackle happens to be the name of a video on demand company. It's a reference to the breakfast cereal of the same name.

Pedantic clarification for the unfamiliar: the breakfast cereal is named Rice Krispies while Snap, Crackle, and Pop are the names of the cartoon mascots on the box.

Re: AWS us-east-2 outage

#246

Post by zob_cloud »

throwaway2016a wrote: post #163
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Architect here. We had an outage and we have a very complete architecture. The issue is, the services were still reachable via internal health checks. So instead of taking the effected servers out of service they stayed in. We had to resolve it by manually shutting down all the servers in the affected AZ. Which is normally not needed. There are of course a lot of companies that aren't architected with multi-AZ at all…

Which internal health checks are you referring to?

Re: AWS us-east-2 outage

#247

Post by throwaway2016a »

zob_cloud wrote: post #246
throwaway2016a wrote: post #163

Earlier quoted context omitted.

Architect here. We had an outage and we have a very complete architecture. The issue is, the services were still reachable via internal health checks. So instead of taking the effected servers out of service they stayed in. We had to resolve it by manually shutting down all the servers in the affected AZ. Which is normally not needed. There are of course a lot of companies that aren't architected with multi-AZ at all…

Which internal health checks are you referring to?

Both the EC2 instance health and our HTTP health checks. If either of those failed the server would have been removed from the load balancer, but they didn't fail.

Only the external health checks that hit the system from an outside service were failing. And because those spread out the load across the AZs, only a fraction of them were failing and no good way to tell the pattern of failure.

I did have some Kubernetes pods become unhealthy but only because they relied on making calls to servers that were in a different AZ.

Re: AWS us-east-2 outage

#248

Post by danw1979 »

eurasiantiger wrote: post #62
danw1979 wrote: post #50

Earlier quoted context omitted.

Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.

Thanks, this comment made it very clear to me that I never want to touch a terraform system.

Nothing is perfect, there’s probably good reason for this behaviour … but it is rarely something that happens anyway. and you know, deleting a key for the state lock (one that explicitly tells you when and who created it) ain’t that hard or a that big of a deal.

Re: AWS us-east-2 outage

#249

Post by linsomniac »

_xnmw wrote: post #146
linsomniac wrote: post #78

I understand that us-east is AWS's oldest and biggest facility, but Amazon seems to have more money than Croesus, why aren't they fixing/rebuilding/replacing us-east with something more modern?

Because money can't fix everything? In fact sometimes having too much money makes it worse, as YC startup wisdom says.

I've avoided responding since my reply that started this was downvoted... But...

Agreed, money can't solve everything. BUT, this seems like an extremely solvable problem. That's why I'm so surprised.

Re: AWS us-east-2 outage

#250

Post by Nextgrid »

outworlder wrote: post #180
dmalvarado wrote: post #126

I dunno how else to put it. Having EVERYTHING on AWS is a national security threat. This isn't good, and someone who can do something about it needs to.

Yeah, let's place everything in large colos instead. Those never fail, right?

But the colos aren't usually managed by a single control plane controlled by a single company, so while they can all fail, they will generally do so independently.
Post reply on HN