Earlier quoted context omitted.
Isn't this kind of circular dependency what lead to extended downtime a while back?
It reminds me of facebook. Staff was locked out of the office due to the outage they were supposed to fix.
[deleted]
AWS North Virginia data center outage – resolved
171–180 of 214 posts
Re: AWS North Virginia data center outage – resolved
#172Earlier quoted context omitted.
Isn't this kind of circular dependency what lead to extended downtime a while back?
It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.
a circular dependency and a single point of failure are not the same thing. If I have a single point of failure and it is down, I fix that and things work again. If I have circular dependency, there is no obvious way to fix anything that is broken any longer.
Re: AWS North Virginia data center outage – resolved
#173Earlier quoted context omitted.
People say this, but this this was just a single AZ, and in the last 3 years of running my startup mostly out of use-1, and we've only had one regional outage, and even that was partial, with most instances uneffected. And honestly, everybody else's stuff is in use-1, so at least your failures are correlated with your customers lol.
>And honestly, everybody else's stuff is in use-1 Yeah, but why put your eggs in that basket? I moved all our services from east to west/oregon a decade ago and haven't looked back.
Sometimes you need capacity and you have to choose where the capacity is not where you would like it to be. Unfortunately, the days of cloud bursting, and thinking of the cloud as an unlimited resource where you can spin up and spin down machines at will is vanishing. Power availability and supply chain lead times combined with unprecedented demand are the reason for this. That's why you see all the hyperscalers recently reporting on their "backlog" in their earnings reports.
Re: AWS North Virginia data center outage – resolved
#174In this thread: People confusing AZs, Regions, and why people would want to use single AZs.
Re: AWS North Virginia data center outage – resolved
#175Earlier quoted context omitted.
I could totally get into “Ops Thriller” genre of novels like this.
there are dozens of us!
There are often little bits of Neal Stephenson or Andy Weir novels which sound a little like this, describing a technical fault in a plot-driven way (often as a cascade), and I do find those to be uniquely enjoyable. I'm sure there are other authors who do similar things, though maybe "cloud/AI data center" stories should be its own micro-genre, given how crucial these things are to society.
Re: AWS North Virginia data center outage – resolved
#176Earlier quoted context omitted.
The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…
Anyone who thinks one cloud provider will provide them full resilience is fooling themselves. You need multicloud for true high availability. But then you want to use the same stack across providers and all the proprietary technologies (even hidden from you with things like terraform) are suddenly loosing their luster.
I don’t think any actually believes that.
What people usually think is “resilience up to a reasonable level of risk and cost”.
Multi-cloud is simply isn’t cost beneficial for 99.9% of problems.
And for a lot of businesses who talk about risk, saying “we followed AWS best practices but AWS went down” is an acceptable answer to the question of liability.
If you are in a position where AWS going down is a reasonable risk, then you’re already in a specialised enough domain to have engineers who understand how to deliver HA across different vendors.
Re: AWS North Virginia data center outage – resolved
#177Earlier quoted context omitted.
The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…
IAM isn’t even really the most painful dependency. Route53 is. The control plane only runs out of use1. Better make sure the only DNS operations you run during an outage are data plane queries and health check failovers.
Is there an architectural reason it’s not for replicas in the other AZs?
Re: AWS North Virginia data center outage – resolved
#178These things are dangerous. Someone who can take AWS down such as an employee can place a bet. These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.
It's a good thing big tech hires for ethical engineers and not ones that only care about money or social status.
Leaders with a vision for all of us.
Re: AWS North Virginia data center outage – resolved
#179Earlier quoted context omitted.
The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…
> outside of China [Nitpick] There are a few more AWS partitions like GovCloud: https://jasonbutz.info/2023/07/aws-partitions/
Since this article was written, AWS also added European Sovereign Cloud as a partition: aws-eusc.
Re: AWS North Virginia data center outage – resolved
#180Earlier quoted context omitted.
The idea that AWS's services are fully regionalized or isolated has always been a myth. All the identity and access services for the public cloud outside of China (aka "IAM for the aws partition" to employees) are centralized in us-east-1. This centralization is essentially necessary in order to have a cohesive view of an account, its billing, and its permissions. And IAM is not a wholly independent software stack: t…
Anyone who thinks one cloud provider will provide them full resilience is fooling themselves. You need multicloud for true high availability. But then you want to use the same stack across providers and all the proprietary technologies (even hidden from you with things like terraform) are suddenly loosing their luster.
I jest, Anyone who thinks multicloud will provide them full resilience is fooling themselves. You need colocated hardware for true high availability.