Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

261–270 of 1001 posts

Re: AWS multiple services outage in us-east-1

#262

So, uh, over the weekend I decided to use the fact that my company needs a status checker/page to try out Elixir + Phoenix LiveView, and just now I found out my region is down while tinkering with it and watching Final Destination. That’s a little too on the nose for my comfort.

Well at least you don't have to figure out how to test your setup locally.

Re: AWS multiple services outage in us-east-1

#264
post #46

Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!

Doing pretty well up here in Tokyo region for now! Just can't log into console and some other stuff.

Doesn't this mean you are not regionally isolated from us-east-1?

Re: AWS multiple services outage in us-east-1

#266

Just a couple of days ago in this HN thread [0] there were quite some users claiming Hetzner is not an options as their uptime isn't as good as AWS, hence the higher AWS pricing is worth the investment. Oh, the irony. [0]: https://news.ycombinator.com/item?id=45614922

I work at a small / medium company with about ~20 dedicated servers and ~30 cloud servers at Hetzner. Outages have happened, but we were lucky that the few times it did happen, it was never a problem / actual downtime.

One thing to note is that there are some scheduled maintenances were we needed to react.

Re: AWS multiple services outage in us-east-1

#267
post #233

“Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.” It’s always DNS.

Even when it's not DNS, it's DNS.

Re: AWS multiple services outage in us-east-1

#268

It's fun watching their list of "Affected Services" grow literally in front of your eyes as they figure out how many things have this dependency. It's still missing the one that earned me a phone call from a client.

It's seemingly everything. SES was the first one that I noticed, but from what I can tell, all services are impacted.

[deleted]

Re: AWS multiple services outage in us-east-1

#269

Their status page ( https://health.aws.amazon.com/health/status ) says the only disrupted service is DynamoDB, but it's impacting 37 other services. It is amazing to see how big a blast radius a single service can have.

The same page now says 58 services - just 23 minutes after your post. Seems this is becoming a larger issue.

Re: AWS multiple services outage in us-east-1

#270

Looks like it affected Vercel, too. https://www.vercel-status.com/ My website is down :( (EDIT: website is back up, hooray)

Have you done anything for it to be back up? Looks like mines are still down.

mines are generally down
Post reply on HN