AWS multiple services outage in us-east-1 • Hacker News Board

Re: AWS multiple services outage in us-east-1

#351

Post by YetAnotherNick » Mon, Oct 20, 2025, 10:03 AM UTC

littlecranky67 wrote: post #232

Just a couple of days ago in this HN thread [0] there were quite some users claiming Hetzner is not an options as their uptime isn't as good as AWS, hence the higher AWS pricing is worth the investment. Oh, the irony. [0]: https://news.ycombinator.com/item?id=45614922

Stop making things up. As someone who commented on the thread in favour of AWS, there is almost no mention of better uptime in any comment I could find.

I could find one or two downvoted or heavily critisized comments, but I can find more people mentioning the opposite.

Re: AWS multiple services outage in us-east-1

#352

Post by rvitorper » Mon, Oct 20, 2025, 10:04 AM UTC

JCM9 wrote: post #336

US-East-1 is literally the Achilles Heel of the Internet.

Exactly

Re: AWS multiple services outage in us-east-1

#353

Post by jacquesm » Mon, Oct 20, 2025, 10:05 AM UTC

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?'

Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically eroding that strength.

Re: AWS multiple services outage in us-east-1

#354

Post by greatgib » Mon, Oct 20, 2025, 10:05 AM UTC

When I follow the link, I arrive on a "You broke reddit" page :-o

Re: AWS multiple services outage in us-east-1

#355

Post by miyuru » Mon, Oct 20, 2025, 10:06 AM UTC

polaris64 wrote: post #229

It looks like DNS has been restored: dynamodb.us-east-1.amazonaws.com. 5 IN A 3.218.182.189

I wonder if the new endpoint was affected as well.

dynamodb.us-east-1.api.aws

Re: AWS multiple services outage in us-east-1

#356

Post by us0r » Mon, Oct 20, 2025, 10:07 AM UTC

stepri wrote: post #233

“Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.” It’s always DNS.

Or expired domains which I suppose is related?

Re: AWS multiple services outage in us-east-1

#357

Post by tsimionescu » Mon, Oct 20, 2025, 10:07 AM UTC

froobius wrote: post #211

fairity wrote: post #183

As this incident unfolds, what’s the best way to estimate how many additional hours it’s likely to last? My intuition is that the expected remaining duration increases the longer the outage persists, but that would ultimately depend on the historical distribution of similar incidents. Is that kind of data available anywhere?

Yes, with no prior knowledge the mathematically correct estimate is: time left = time so far But as you note prior knowledge will enable a better guess.

Note that this is equivalent to saying "there's no way to know". This guess doesn't give any insight, it's just the function that happens to minimize the total expected error for an unknowable duration.

Edit: I should add that, more specifically, this is a property of the uniform distribution, it applies to any event for which EndsAfter(t) is uniformly distributed over all t > 0.

Re: AWS multiple services outage in us-east-1

#358

Post by jwr » Mon, Oct 20, 2025, 10:08 AM UTC

littlecranky67 wrote: post #232

Just a couple of days ago in this HN thread [0] there were quite some users claiming Hetzner is not an options as their uptime isn't as good as AWS, hence the higher AWS pricing is worth the investment. Oh, the irony. [0]: https://news.ycombinator.com/item?id=45614922

As a data point, I've been running stuff at Hetzner for 10 years now, in two datacenters (physical servers). There were brief network outages when they replaced networking equipment, and exactly ONE outage for hardware replacement, scheduled weeks in advance, with a 4-hour window and around 1-2h duration.

It's just a single data point, but for me that's a pretty good record.

It's not because Hetzner is miraculously better at infrastructure, it's because physical servers are way simpler than the extremely complex software and networking systems that AWS provides.

Re: AWS multiple services outage in us-east-1

#359

Post by JCM9 » Mon, Oct 20, 2025, 10:09 AM UTC

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere.

The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Re: AWS multiple services outage in us-east-1

#360

Post by hvb2 » Mon, Oct 20, 2025, 10:09 AM UTC

jacquesm wrote: post #353

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…

> The internet got its main strengths from the fact that it was completely decentralized.

Decentralized in terms of many companies making up the internet. Yes we've seen heavy consolidation in now having less than 10 companies make up the bulk of the internet.

The problem here isn't caused by companies chosing one cloud provider over the other. It's the economies of scale leading us to few large companies in any sector.