Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS multiple services outage in us-east-1

health.aws.amazon.com

Post reply on HN
Print view View original

851–860 of 1001 posts

Re: AWS multiple services outage in us-east-1

#851

Post by coldcache »

ta1243 wrote: post #473
nikolay wrote: post #46

Choosing us-east-1 as your primary region is good, because when you're down, everybody's down, too. You don't get this luxury with other US regions!

It took me so long to realise this is what's important in enterprise. Uptime isn't important, being able to blame someone else is what's important. If you're down for 5 minutes a year because one of your employees broke something, that's your fault, and the blame passes down through the CTO. If you're down for 5 hours a year but this affected other companies too, it's not your fault From AWS to Crowdstrike - system r…

“No one ever got fired for hiring IBM”

Re: AWS multiple services outage in us-east-1

#852

Post by jdlyga »

tonymet wrote: post #826
0x5345414e wrote: post #787

This is having a direct impact on my wellbeing. I was at Whole Foods in Hudson Yards NYC and I couldn’t get the prime discount on my chocolate bar because the system isn’t working. Decided not to get the chocolate bar. Now my chocolate levels are way too low.

"alexa turn on coffee pot" stopped working this morning, and I'm going bonkers.

Alexa is super buggy now anyway. I switched my Echo Dot to Alexa+, and it fails turning on and off my Samsung TV all the time now. You usually have to do it twice.

Re: AWS multiple services outage in us-east-1

#853

Post by nettlin »

melozo wrote: post #777

Even internal Amazon tooling is impacted greatly - including the internal ticketing platform which is making collaboration impossible during the outage. Amazon is incapable of building multi-region services internally. The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform. Makes me wonder how much juice this company has left.

> The Amazon retail site seems available, but I’m curious if it’s even using native AWS or is still on the old internal compute platform.

Some parts of amazon.com seem to be affected by the outage (e.g. product search: https://x.com/wongmjane/status/1980318933925392719)

Re: AWS multiple services outage in us-east-1

#854

Post by oxfordmale »

markus_zhang wrote: post #743
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

It has been quite a while, wondering how many 9s are dropped. 365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.

But is is a partial outage only, so it doesn't count. If you retry a million times everything still works /s

Re: AWS multiple services outage in us-east-1

#855

Post by baubino »

indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

Basic services at my worksite have been offline for almost 8 hours now (things were just glitchy for about 4 hours before that). This is nuts.

Re: AWS multiple services outage in us-east-1

#857

Post by CobrastanJorji »

radium3d wrote: post #827
JCM9 wrote: post #359

Have a meeting today with our AWS account team about how we’re no longer going to be “All in on AWS” as we diversify workloads away. Was mostly about the pace of innovation on core services slowing and AWS being too far behind on AI services so we’re buying those from elsewhere. The AWS team keeps touting the rock solid reliability of AWS as a reason why we shouldn’t diversify our cloud. Should be a fun meeting!

Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at some point. Get your refund and carry on. It's less work for the same outcome.

Multi-cloud. It's fairly unlikely that AWS and Google Cloud are going to fail at the same time.

Re: AWS multiple services outage in us-east-1

#858

Post by bartread »

napolux wrote: post #807
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

worst of all: ring alarm unstoppable siren because the app is down and the keyboard was removed by my parents and put "somewhere in the basement".

Is it hard wired? If so, and if the alarm module doesn’t have an internal battery, can you go to the breaker box and turn off the circuit it’s on? You should be able to switch off each breaker in turn until it stops if you don’t know which circuit it’s on.

If it doesn’t stop, that means it has a battery backup. But you can still make life more bearable. Switch off all your breakers (you probably have a master breaker for this), then open up the alarm box and either pull the battery or - if it’s non-removable - take the box off the wall, put it in a sealed container, and put the sealed container somewhere… else. Somewhere you can’t hear it or can barely hear it until the battery runs down.

Meanwhile you can turn the power back on but make sure you’ve taped the bare ends of the alarm power cable, or otherwise electrically insulated them, until you’re able to reinstall it.

Re: AWS multiple services outage in us-east-1

#859

Post by beachy »

markus_zhang wrote: post #743
indoordin0saur wrote: post #670

Seems like major issues are still ongoing. If anything it seems worse than it did ~4 hours ago. For reference I'm a data engineer and it's Redshift and Airflow (AWS managed) that is FUBAR for me.

It has been quite a while, wondering how many 9s are dropped. 365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.

I haven't done any RFP responses for a while but this question always used to make me furious. Our competitors (some of who had had major incidents in the past) claimed 99.99% availability or more, knowing they would never have to prove it, and knowing they were actually 100% until the day they weren't.

We were more honest, and it probably cost us at least once in not getting business.

Post reply on HN