"The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers."
https://health.aws.amazon.com/health/status?path=service-his...
AWS multiple services outage in us-east-1
811–820 of 1001 posts
Re: AWS multiple services outage in us-east-1
#812altavista.com is also down!
Re: AWS multiple services outage in us-east-1
#813Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!
AWS has been in long term decline, most of the platform is just in keeping the lights on mode. Its also why they are behind on AI, alot of would be innovative employees get crushed under red tape and performance management
Good thing they are the biggest investor into Anthropic
Re: AWS multiple services outage in us-east-1
#814I'm not sure if this is directly related, but I've noticed my Apple Music app has stopped working (getting connection error messages). Didn't realize the data for Music was also hosted on AWS, unless this is entirely unrelated? I've restarted my phone and rebooted the app to no avail, so I'm assuming this is the culprit.
Re: AWS multiple services outage in us-east-1
#815Cool, building in resilience seems to have worked. Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted (not sure if it would have been anyway). My control plane is native multi-region, so while it depends on many impacted services it stayed available. Each region runs in isolation. There is data replication at play but failing to replicate to us-east-1 had no impact on other…
> Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted This seems like such a low bar for 2025, but here we are.
You're also betting that CloudFront isn't one of the several AWS services that only works when us-east-1 is up.
Re: AWS multiple services outage in us-east-1
#816From the great Corey Quinn
Ah yes, the great AWS us-east-1 outage.
Half the internet’s on fire, engineers haven’t slept in 18 hours, and every self-styled “resilience thought leader” is already posting:
“This is why you need multi-cloud, powered by our patented observability synergy platform™.”
Shut up, Greg.
Your SaaS product doesn’t fix DNS, you're simply adding another dashboard to watch the world burn in higher definition.
If your first reaction to a widespread outage is “time to drive engagement,” you're working in tragedy tourism. Bet your kids are super proud.
Meanwhile, the real heroes are the SREs duct-taping Route 53 with pure caffeine and spite.
https://www.linkedin.com/posts/coquinn_aws-useast1-cloudcomp...
Re: AWS multiple services outage in us-east-1
#817Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.
I noticed the same thing and it seems to have gotten much worse around 8:55 a.m. Pacific Time.
By the way, Twilio is also down, so all those login SMS verification codes aren’t being delivered right now.
Re: AWS multiple services outage in us-east-1
#818These things happen when profits are the measure everything. Change your provider, but if their number doesn't go up, they wont be reliable. So your complaints matter nothing because "number go up". I remember the good old days of everyone starting a hosting company. We never should have left.
It certainly doesn't take profit for bad planning to happen...
And everybody starting a hosting company is definitely a profit driven activity.
Re: AWS multiple services outage in us-east-1
#819I know there's a lot of anecdotal evidence and some fairly clear explanations for why `us-east-1` can be less reliable. But are there any empirical studies that demonstrate this? Like if I wanted to back up this assumption/claim with data, is there a good link for that, showing that us-east-1 is down a lot more often?
The unreliability claim is driven by two factors. 1. When aws deploys changes they run through a pipeline which pushes change to regions one at a time. Most services start with us-east-1 first. 2. us-east-1 is MASSIVE and considerably larger than the next largest region. There's no public numbers but I wouldn't be surprised if it was 50% of their global capacity. An outage in any other region never hits the news.
> a pipeline which pushes change to regions one at a time
> When AWS deploys updates to its services, deployments to Availability Zones in the same Region are separated in time to prevent correlated failure.
https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...
Re: AWS multiple services outage in us-east-1
#820Do events like this stir conversations in small to medium size businesses to escape the cloud?