Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

911–920 of 1001 posts

Re: AWS multiple services outage in us-east-1

#911

Post by jordanb »

SOLAR_FIELDS wrote: post #880
time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

This reminds me of the time that Google’s Paris data center flooded and caught on fire a few years ago. We weren’t actually hosting compute there, but we were hosting compute in AWS EU datacenter nearby and it just so happened that the dns resolver for our Google services elsewhere happened to be hosted in Paris (or more accurately it routed to Paris first because it was the closest). The temp fix was pretty fun, tha…

Couldn't you just patch your coredns deployment to specify different forwarders?

Re: AWS multiple services outage in us-east-1

#912

Post by raverbashing »

vladvasiliu wrote: post #691
time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

> Identity Center and only put it in us-east-1 Is it possible to have it in multiple regions? Last I checked, it only accepted one region. You needed to remove it first if you wanted to move it.

Security people and ignoring resiliency and failure modes: a tale as old as time

Re: AWS multiple services outage in us-east-1

#913

Post by shepherdjerred »

stevejb wrote: post #873
melozo wrote: post #777

Even internal Amazon tooling is impacted greatly - including the internal ticketing platform which is making collaboration impossible during the outage. Amazon is incapable of building multi-region services internally. The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Makes me wonder how much juice this company has left.

Amazon's revenue in 2024 was about the size of Belgium's GDP. Higher than Sweden or Ireland. It makes a profit similar to Norway, without drilling for offshore oil or maintaining a navy. I think they've got plenty of juice left.

You could argue Amazon's security is an irregular military force

Re: AWS multiple services outage in us-east-1

#914

Post by ashikns »

iammrpayments wrote: post #908
haunter wrote: post #861

The Premier League said there will be only limited VAR today w/o the automatic offside system becasue of the AWS outage. Weird timeline we live in https://www.bbc.com/news/live/c5y8k7k6v1rt?post=asset%3Ad902...

Why is VAR connected to the internet? Are they trying to gather data on offside players customers to improve recommedations?

I worked in a similar system. The raw data from the field first goes to a cloud hosted event queue of some sort, then a database, then back to whatever app/screen on field. The data doesn't just power on-field displays. There's a lot of online websites, etc that needs to pull data from an api.

Re: AWS multiple services outage in us-east-1

#916

Post by afavour »

iammrpayments wrote: post #908
haunter wrote: post #861

The Premier League said there will be only limited VAR today w/o the automatic offside system becasue of the AWS outage. Weird timeline we live in https://www.bbc.com/news/live/c5y8k7k6v1rt?post=asset%3Ad902...

Why is VAR connected to the internet? Are they trying to gather data on offside players customers to improve recommedations?

I wouldn't be at all surprised if people pay for API access to the data. I've worked with live sports data before, it's a very profitable industry to be in when you're the one selling the data.

Of course in a sane world you'd have an internal fallback for when cloud connectivity fails but I'm sure someone looked at the cost and said "eh, what's the worst that could happen?"

Re: AWS multiple services outage in us-east-1

#918

Post by barbazoo »

time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

Wow, you really *have* to exercise the region failover to know if it works, eh? And that confidence gets weaker the longer it’s been since the last failover I imagine too. Thanks for sharing what you learned.

Re: AWS multiple services outage in us-east-1

#919

Post by shdjhdfh »

barbazoo wrote: post #918
time0ut wrote: post #547

Interesting day. I've been on an incident bridge since 3AM. Our systems have mostly recovered now with a few back office stragglers fighting for compute. The biggest miss on our side is that, although we designed a multi-region capable application, we could not run the failover process because our security org migrated us to Identity Center and only put it in us-east-1, hard locking the entire company out of the AWS…

Wow, you really *have* to exercise the region failover to know if it works, eh? And that confidence gets weaker the longer it’s been since the last failover I imagine too. Thanks for sharing what you learned.

You should assume it will not work unless you test it regularly. That's a big part of why having active/active multi-region is attractive, even though it's much more complex.

Re: AWS multiple services outage in us-east-1

#920

Post by TheJoeMan »

nettlin wrote: post #853
melozo wrote: post #777

Even internal Amazon tooling is impacted greatly - including the internal ticketing platform which is making collaboration impossible during the outage. Amazon is incapable of building multi-region services internally. The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Makes me wonder how much juice this company has left.

> The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Some parts of amazon.com seem to be affected by the outage (e.g. product search: https://x.com/wongmjane/status/1980318933925392719 )

Reviewing order history was also spotty. According to my wife “keep hitting refresh getting different dogs each time until it works”
Post reply on HN