There are entire apps like Reddit that are still not working. What the fuck is going on?
AWS multiple services outage in us-east-1
671–680 of 1001 posts
Re: AWS multiple services outage in us-east-1
#672Cool, building in resilience seems to have worked. Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted (not sure if it would have been anyway). My control plane is native multi-region, so while it depends on many impacted services it stayed available. Each region runs in isolation. There is data replication at play but failing to replicate to us-east-1 had no impact on other…
> Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted
This seems like such a low bar for 2025, but here we are.
Re: AWS multiple services outage in us-east-1
#673Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
SEV-0 for my company this morning. We can't connect to RDS anymore.
Re: AWS multiple services outage in us-east-1
#674stupid question: is buying a server rack and running it at home subject to more downtimes in a year than this? has anyone done an actual SLA analysis?
I've been dabbling in this since 1998, It's almost always ISP and power outages that get you. There are ways to mitigate those (primary/secondary ISPs, UPSes, and generators) but typically unless you're in a business district area of a city, you'll just always be subject to problems
Re: AWS multiple services outage in us-east-1
#675Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…
What good is jumping through extraordinary hoops to be multi cloud if docker, netlify, stripe, intercom, npm, etc all go down along with us-east-1?
Re: AWS multiple services outage in us-east-1
#676Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
Yep, confirmed worse - DynamoDB now returning "ServiceUnavailableException"
Re: AWS multiple services outage in us-east-1
#677thundering herd problems.... every time they say they fix it something else breaks
Re: AWS multiple services outage in us-east-1
#678Had a meeting where developers were discussing the infrastructure for an application. A crucial part of the whole flow was completely dependant on an AWS service. I asked if it was a single point of failure. The whole room laughed, I rest my case.
Re: AWS multiple services outage in us-east-1
#679Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
In addition to those, Sagemaker also fails for me with an internal auth error specifically in Virginia. Fun times. Hope they recover by tomorrow.
Re: AWS multiple services outage in us-east-1
#680Whose idea was it to make the whole world dependent on us-east-1?