Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

951–960 of 1001 posts

Re: AWS multiple services outage in us-east-1

#951

Post by testplzignore »

tete wrote: post #540
JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

> The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting! This is and was never true. I've done setups in the past where monitoring happened "multi cloud" with also multiple dedicated servers. Was pretty broad so you could actually see where things broke. Was quite some time ago so I don't have the data, but AWS never came out on top. It…

Netcraft confirmed it? I haven't heard that name since the Slashdot era :)

Re: AWS multiple services outage in us-east-1

#952

Post by ronakjain90 »

we[1] operate out of `us-east-1` but chose to not use any of the cloud based vendor lockin (sorry vercel, supabase, firebase, planetscale etc). Rather a few droplets in DigitalOcean(us-east-1) and Hetzner(eu). We serve 100 million requests/mo, few million user generated content(images)/mo at monthly cost of just about $1000/mo.

It's not difficult, it's just that we engineers chose convenience and delegated uptime to someone else.

[1] - https://usetrmnl.com

Re: AWS multiple services outage in us-east-1

#953

Post by procaryote »

markus_zhang wrote: post #743
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

It has been quite a while, wondering how many 9s are dropped. 365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.

it's a matter of perspective... 9.9999% is real easy

Re: AWS multiple services outage in us-east-1

#954

Post by ct_list »

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

[dead]

Re: AWS multiple services outage in us-east-1

#955

Post by TZubiri »

0x5345414e wrote: post #787

This is having a direct impact on my wellbeing. I was at Whole Foods in Hudson Yards NYC and I couldn’t get the prime discount on my chocolate bar because the system isn’t working. Decided not to get the chocolate bar. Now my chocolate levels are way too low.

That's it, internet centralization has gone too far, call your congress(wo)man

Re: AWS multiple services outage in us-east-1

#956

Post by lkjdsklf »

nijave wrote: post #899
stepri wrote: post #233

“Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.” It’s always DNS.

I don't think that's necessarily true. The outage updates later identified failing network load balancers as the cause--I think DNS was just a symptom of the root cause I suppose it's possible DNS broke health checks but it seems more likely to be the other way around imo

I don’t work for aws, but a different cloud provider so this is not a description of this incident, but an example of the kind of thing that can happen

One particular “dns” issue that caused an outage was actually a bug in software that monitors healthchecks.

It would actively monitor all servers for a particular service (by updating itself based on what was deployed) and update dns based on those checks.

So when the health check monitors failed, servers would get removed from dns within a few milliseconds.

Bug gets deployed to health check service. All of a sudden users can’t resolve dns names because everything is marked as unhealthy and removed from dns.

So not really a “dns” issue, but it looks like one to users

Re: AWS multiple services outage in us-east-1

#957

Post by jonplackett »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

The problem is now that, what’s anyone going to do? Leave?

I remember a meme years ago about Nestle. It was something like: GO ON, BOYCOT US - I BET YOU CAN’T - WE MAKE EVERYTHING.

Same meme would work for Aws today.

Re: AWS multiple services outage in us-east-1

#958

Post by MaKey »

jonplackett wrote: post #957
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

The problem is now that, what’s anyone going to do? Leave? I remember a meme years ago about Nestle. It was something like: GO ON, BOYCOT US - I BET YOU CAN’T - WE MAKE EVERYTHING. Same meme would work for Aws today.

> Same meme would work for Aws today.

Not really, there are enough alternatives.

Re: AWS multiple services outage in us-east-1

#960

Post by saltserv »

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

[dead]
Post reply on HN