Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS North Virginia data center outage – resolved

cnbc.com

Post reply on HN
Print view View original

141–150 of 214 posts

Re: AWS North Virginia data center outage – resolved

#141

Post by Eridrus »

linsomniac wrote: post #109
Eridrus wrote: post #98

Earlier quoted context omitted.

People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.

>And honestly, everybody else's stuff is in use-1 Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.

90% of customers are located in use-1. Latency to use-1 is more important than being up when everyone else is down.

Re: AWS North Virginia data center outage – resolved

#142

Post by mimischi »

altern8 wrote: post #138
tornikeo wrote: post #133

I wonder if hetzner had better uptime in EU than AWS this year.

Why no love for OVH? I find Hetzner's UI to be super-confusing, making it hard to manage things.

At the rate that people claim AWS us-east to go down, folks will argue that OVH has a tendency to go up in flames!

Re: AWS North Virginia data center outage – resolved

#143

Post by echelon_musk »

999900000999 wrote: post #75
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

Too many people are using it. In fantasy magic dream land loads are distributed evenly across different cloud providers. A single point of failure doesn't exist. It worked out with my first girlfriend. The twins are fluent in English and Korean. They know when deploying a large scale service to not only depends on AWS. Healthcare in the US is affordable. All types of magical stuff exist here. But no. It's another day…

> It worked out with my first girlfriend. The twins are fluent in English and Korean.

You were dating twins as a form of redundancy?!

Re: AWS North Virginia data center outage – resolved

#144

Post by JackSlateur »

999900000999 wrote: post #88
afro88 wrote: post #84

Earlier quoted context omitted.

Core AWS services use it too. Even if you are hosted in another region, you can still be affected by a US-East 1 outage

The idea would be to actually load distribute between different cloud providers. But even then , the load balancer needs to run somewhere. Which becomes a new single point of failure. I’m sure someone smarter than me has figured this out.

DNS

Re: AWS North Virginia data center outage – resolved

#146

Post by rswail »

So in the comments here we have the usual about us-east-1, it's centralized, it's a SPOF for AWS, they should fix it, don't put your stuff there, etc.

This was one data centre in one zone of a multi-zone region.

Yes IAM/R53 and others are centralized there, yes, reworking those service to be decentralized and cross-region would be a Good Thing. But us-east-1 is already multi-zone (6 with a seventh marked as "coming in 2026") with multi DC within zones. From memory, when a global service like IAM is out, it's more likely to be bugs in the implementation or dependency than a "if this was cross-region it wouldn't have died" issue.

But this wasn't an outage of any AWS global service this time. The only one that seemed to have more impact was/is MSK. Which is likely to be more of an issue with Kafka than anything AWS related.

Re: AWS North Virginia data center outage – resolved

#148

Post by zaphirplane »

dlenski wrote: post #101
cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

Services outside of us-east-1 don’t call us-east-1 for IAM data plane thou right ?

Re: AWS North Virginia data center outage – resolved

#149

Post by zaphirplane »

stephenr wrote: post #128
dlenski wrote: post #101

Earlier quoted context omitted.

The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…

> And IAM is not a wholly independent software stack: they rely on DynamoDB and a few other services, which in turn have a circular dependency on IAM. When you dogfood your own Rube Goldberg machine.

We should let the IAM service team know if this glaring gap the hn thread figured out /s

I’m 99% ;) certain dependencies of foundational services are a well discussed topic

Re: AWS North Virginia data center outage – resolved

#150

Post by ohnei »

cmiles8 wrote: post #34

AWS’s US-East 1 continues to be the Achilles heel of the Internet. And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

I've always been impressed by Amazon's ability to present the shittiest experience possible and imply the blame is with things like isolation that they don't really provide.
Post reply on HN