It's fun watching their list of "Affected Services" grow literally in front of your eyes as they figure out how many things have this dependency.
It's still missing the one that earned me a phone call from a client.
AWS multiple services outage in us-east-1
151–160 of 1001 posts
Re: AWS multiple services outage in us-east-1
#152Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!
Sometimes we all need a tech shutdown.
As they say, every cloud outage has a silver lining.
* Give the computers a rest, they probably need it. Heck, maybe the Internet should just shut down in the evening so everyone can go to bed (ignoring those pesky timezone differences)
* Free chaos engineering at the cloud provider region scale, except you didn't opt in to this one and know about in advance, making it extra effective
* Quickly figure out a map which of the things you use have a dependency on a single AWS region without no capability to change or re-route
Re: AWS multiple services outage in us-east-1
#153Meanwhile my pair of 12 year old raspberry pi's hangling my home services like DNS survive their 3rd AWS us-east-1 outage. "But you can't do webscale uptime on your own" Sure. I suspect even a single pi with auto-updates on has less downtime.
[deleted]
Re: AWS multiple services outage in us-east-1
#154Clearly this is all some sort of mass delusion event, the Amazon Ring status says everything is working. https://status.ring.com/ (Useless service status pages are incredibly annoying)
Atlassian is down as well so they probably can't access their Atlassian Statuspage admin panel to update it.
Re: AWS multiple services outage in us-east-1
#155Internet, out. Very big day for an engineering team indeed. Can't vibe code your way out of this issue...
> Can't vibe code your way out of this issue...
Exactly. This time, some LLM providers are also down and can't help vibe coders on this issue.
Re: AWS multiple services outage in us-east-1
#156Re: AWS multiple services outage in us-east-1
#157Looks like we're back!
Re: AWS multiple services outage in us-east-1
#158Internet, out. Very big day for an engineering team indeed. Can't vibe code your way out of this issue...
Pour one out for everyone on-call right now.
Re: AWS multiple services outage in us-east-1
#159I find it interesting that AWS services appear to be so tightly integrated that when there's an issue in a region, it affects most or all services. Kind of defeats the purported resiliency of cloud services.
I think a lot of its probably technical debt. So much internally still relies on legacy systems in US-East-1, and every time this happens I'm sure theres a discussion internally about decoupling that reliance which then turns into a massive diagram that looks like a family tree dating back a thousand years of all the things that need to be changed to stop it happening.
Re: AWS multiple services outage in us-east-1
#160But but this is a cloud, it should exist in the cloud.