Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

401–410 of 1001 posts

Re: AWS multiple services outage in us-east-1

#401

Post by myself248 »

shawn_w wrote: post #335

One of the radio stations I listen to is just dead air tonight. I assume this is the cause.

A physical on-air broadcast station, not a web stream? That likely violates a license; they're required to perform station identification on a regular basis.

Of course if they had on-site staff it wouldn't be an issue (worst case, just walk down to the transmitter hut and use the transmitter's aux input, which is there specifically for backup operations like this), but consolidation and enshittification of broadcast media mean there's probably nobody physically present.

Re: AWS multiple services outage in us-east-1

#402

Post by mlrtime »

padjo wrote: post #369
jacquesm wrote: post #353

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

>Yes it does happen but very rarely to the tune of a few hours every 5-10 years.

It is rare, but it happens at LEAST 2-3x a year. AWS us-east-1 has a major incident affecting multiple services (that affect most downstream aws services) multiple times a year. Usually never the same root cause.

Not very many people realize that there are some services that still run only in us-east-1.

Re: AWS multiple services outage in us-east-1

#403

Post by seydor »

fairity wrote: post #183

As this incident unfolds, what’s the best way to estimate how many additional hours it’s likely to last? My intuition is that the expected remaining duration increases the longer the outage persists, but that would ultimately depend on the historical distribution of similar incidents. Is that kind of data available anywhere?

1440 min

Re: AWS multiple services outage in us-east-1

#404

Post by JCM9 »

goinggetthem wrote: post #224

This is from Amazon's latest earnings call when Andy Jessy was asked why they aren't growing as much as there competitors "I think if you look at what matters to customers, what they care they care a lot about what the operational performance is, you know, what the availability is, what the durability is, what the latency and throughput is of of the various services. And I think we have a pretty significant advantage…

Well that aged well

Re: AWS multiple services outage in us-east-1

#405

Post by bartvk »

ctbellmar wrote: post #75

Various AI services (e.g. Perplexity) are down as well

I don't like how they phrased it. From the Verge:

“Perplexity is down right now,” Perplexity CEO Aravind Srinivas said on X. “The root cause is an AWS issue. We’re working on resolving it.”

What he should have said, IMHO, is "The root cause is that Perplexity fully depends on AWS."

I wonder if they're actually working on resolving that, or that they're just waiting for AWS to come back up.

Re: AWS multiple services outage in us-east-1

#406

Post by kkfx »

Honestly anyone do have outages, that's nothing extraordinary, what's wrong is the number of impacted services. We choose (at least almost choose) to ditch mainframes for clusters also for resilience. Now with cheap desktop iron labeled "stable enough to be a serious server" we have seen mainframes re-created sometimes with a cluster of VM on top of a single server, sometimes with cloud services.

Ladies and Gentleman's it's about time to learn reshoring in the IT world as well. Owning nothing, renting all means extreme fragility.

Re: AWS multiple services outage in us-east-1

#409

Post by cmiles8 »

JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

This. When Andy Jassy got challenged by analysts on the last earnings call on why AWS has fallen so far behind on innovation in areas his answer was a hand wavy response that diverted attention to say AWS is durable, stable, and reliable and customers care more about that. Oops.

Re: AWS multiple services outage in us-east-1

#410

Post by mlrtime »

kalleboo wrote: post #151

It's fun watching their list of "Affected Services" grow literally in front of your eyes as they figure out how many things have this dependency. It's still missing the one that earned me a phone call from a client.

When these major issues come up, all they have is symptoms and not causes. Maybe not until the dynamo oncall comes on and says its down, then everyone knows at least the reason for their teams outage.

The scale here is so large they don't know the complete dependency tree until teams check-in on what is out or not, growing this list. Of course most of it is automated, but getting on 'Affected Services' is not.

Post reply on HN