Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

801–810 of 1001 posts

Re: AWS multiple services outage in us-east-1

#801

Post by zild3d »

mlhpdx wrote: post #637

Cool, building in resilience seems to have worked. Our static site has origins in multiple regions via CloudFront and didn’t seem to be impacted (not sure if it would have been anyway). My control plane is native multi-region, so while it depends on many impacted services it stayed available. Each region runs in isolation. There is data replication at play but failing to replicate to us-east-1 had no impact on other…

active/active? curious what the data stack looks like as that tends to be the hard part

Re: AWS multiple services outage in us-east-1

#802

Post by bongodongobob »

jacquesm wrote: post #353

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…

You simply cannot avoid it. There are so many applications and services that use AWS. Companies cant sit on 100% in-house software stacks.

Re: AWS multiple services outage in us-east-1

#803

Post by melozo »

willsmith72 wrote: post #793
melozo wrote: post #777

Even internal Amazon tooling is impacted greatly - including the internal ticketing platform which is making collaboration impossible during the outage. Amazon is incapable of building multi-region services internally. The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Makes me wonder how much juice this company has left.

It seems reasonable to me that Amazon (retail) would build better AZ redundancy into their services than say Snapchat or a bank

Sure, but it’s not reasonable that internal collaboration platforms built for ticketing engineers about outages doesn’t work during the outage. That would be something worth making multi-region at a minimum.

Re: AWS multiple services outage in us-east-1

#805

Post by davedx »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Andy Jassy is the Tim Cook of Amazon

Rest and vest CEOs

Re: AWS multiple services outage in us-east-1

#806

Post by dvsgaevsvsgavsv »

altbdoor wrote: post #678

Had a meeting where developers were discussing the infrastructure for an application. A crucial part of the whole flow was completely dependant on an AWS service. I asked if it was a single point of failure. The whole room laughed, I rest my case.

Amazing. So you will build your own load balancer that sends loads between AWS and Gcloud and make it the single point of failure instewd?

Re: AWS multiple services outage in us-east-1

#807

Post by napolux »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

worst of all: ring alarm unstoppable siren because the app is down and the keyboard was removed by my parents and put "somewhere in the basement".

Re: AWS multiple services outage in us-east-1

#808

Post by Tuna-Fish »

glemmaPaul wrote: post #193

LOL making one db service a central point of failure, charge gold for small compute instances. Rage about needing Multi AZ, make the costs come onto the developer/organization. But, now fail on a region level, so are we going to now have multi-country setup for simple small applications?

It's not DNS

There's no way it's DNS

It was DNS

Re: AWS multiple services outage in us-east-1

#809

Post by belter »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

This looks like one their worst outage in 15 years and us-east-1 still shows as degraded but I had no outages, as dont use us-east-1. Are you seeing issues on other regions?

https://health.aws.amazon.com/health/status?path=open-issues

The closest to their identification of a root cause seems to be this one:

"Oct 20 8:43 AM PDT We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations."

Post reply on HN