Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-2 outage

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 254 posts

Re: AWS us-east-2 outage

#191

Post by h02 »

throwaway2016a wrote: post #163
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Architect here. We had an outage and we have a very complete architecture. The issue is, the services were still reachable via internal health checks. So instead of taking the effected servers out of service they stayed in. We had to resolve it by manually shutting down all the servers in the affected AZ. Which is normally not needed. There are of course a lot of companies that aren't architected with multi-AZ at all…

Yup, exact same here. All of the multi-AZ failover depends on AWS recognizing that their AZ is having an issue, and they never reported having an issue on any health-check so no failover ever happened. We started being able to make progress when AWS told us which AZ was having issues. It still took some time for us to manually shift away from that AZ (manually promoting ElastiCache replicas to primary, switching RDS clusters around, etc.) because all of the AWS failover functionality did not function as they should have and we were relying on that. Multi-region failover would have made us more fault tolerant but our infrastructure wasn't setup for that yet (besides an RDS failover in a separate region). Here's to hoping we never have a Route53 or global AWS API Gateway failure! Then even multi-region will not do us much good. Perhaps we should have some backup servers on the moon, then in case of nuclear warfare we can still be online via satellite.

P.S. AWS has said they have resolved the issue for almost 2 hours now and we are still having issues with us-east-2a.

Re: AWS us-east-2 outage

#192

Post by corobo »

dang wrote: post #41
corobo wrote: post #9

Ahh Always check HN before trying to diagnose weird issues that shouldn't be connected

Living a bit more dangerously at the moment as HN is still running temporarily on AWS. (I'd link to the threads about this from a few weeks ago but am on my phone ATM.)

I did notice it being a little slow but I'm also on 4G at the moment (it got the blame)

Re: AWS us-east-2 outage

#194

Post by laumars »

throwawaymaths wrote: post #71
alfalfasprout wrote: post #66

Earlier quoted context omitted.

Interesting to see it's been a loss of power that caused this. Usually the better datacenters have multiple levels of power redundancy including emergency backup generators.

Insert clip of O'Brien explaining to cardassians why there are backups for backups

In case anyone is unaware of the reference, that’s taken from Star Trek Deep Space 9 https://youtu.be/UaPkSU8DNfY

Re: AWS us-east-2 outage

#195

Post by happymellon »

blamarvt wrote: post #165
happymellon wrote: post #154

Earlier quoted context omitted.

Considering almost all of the services are multi-zone, it's not hard to add in a couple of lines to make them resilient against this. People are just unaware, and probably making bad calls in the name of being "portable".

If your application and infra can magically utilize multiple zones with “a couple lines”… then I would say you are miles ahead of just about every other web company.

> you are miles ahead of just about every other web company.

I'm curious who these web companies are.

Use something like Lambda and you get multi-az for free.

https://docs.aws.amazon.com/lambda/latest/dg/security-resili...

Dynamo is another service that wouldn't be impacted as it is multi-az.

Getting postgres RDS multi-region would require the extra couple of lines in your CDK, but is fairly straightforward.

Re: AWS us-east-2 outage

#196

Post by happymellon »

the_fury wrote: post #175
happymellon wrote: post #154

Earlier quoted context omitted.

Considering almost all of the services are multi-zone, it's not hard to add in a couple of lines to make them resilient against this. People are just unaware, and probably making bad calls in the name of being "portable".

Or how about "I'm fully aware, I've done the math taking into account both cost and complexity of implementation and cost of downtime, and I'm probably making fantastic calls based on my actual needs."

If you had "done the math" then you would have gone serverless and gained multi-az for free, as it is almost always the cheapest option.

Re: AWS us-east-2 outage

#197

Post by mhb »

latchkey wrote: post #107
mhb wrote: post #98

Earlier quoted context omitted.

Given the scope of the effort invested in attempting to prevent duck and goose crap on the world's docks, I'm skeptical that this tactic is effective.

Shrug... the datacenter is land locked (different animal species) and the problem hasn't happened again in multiple years. I think you're taking the Eagle a bit too seriously though... if we didn't do anything how would we know? It isn't like this was an expensive thing to try out.

OK. It's just that I am one of those people who have tried to solve the duck/goose problem and would be delighted if a fake eagle or owl worked.

Re: AWS us-east-2 outage

#199

Post by gamblor956 »

Nextgrid wrote: post #160
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

I have 2 takes on this: 1) AWS is already really expensive, just on a single AZ. Replicating to a second AZ would almost double your costs. I can't help but bring up the point that an old-school bare-metal setup on something like Hetzner/OVH/etc becomes significantly more cost-effective since you're not using AWS's advantages in this area anyway (and as we've seen in practice, AWS is nowhere near more reliable - how…

Going bare-metal is a premature optimization. Most startups that go that route don't survive long enough to make use of this optimization.

Take advantage of AWS (or Azure, or DO) until you're big enough that bringing the action in-house is a financially and technically prudent option.

Re: AWS us-east-2 outage

#200

Post by jozzy-james »

throwaway2016a wrote: post #163
poxrud wrote: post #97

The fact that so many popular sites/services are experiencing issues due to a single AZ failure makes me think that there is a serious shortage of good cloud architects/engineers in the industry. It would be one thing if this was a Regional failure, but a single AZ failure should not have any noticeable effect.

Architect here. We had an outage and we have a very complete architecture. The issue is, the services were still reachable via internal health checks. So instead of taking the effected servers out of service they stayed in. We had to resolve it by manually shutting down all the servers in the affected AZ. Which is normally not needed. There are of course a lot of companies that aren't architected with multi-AZ at all…

that tracks with our experience as well
Post reply on HN