Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

551–560 of 1001 posts

Re: AWS us-east-1 outage

#551

Post by kingcharles »

dangrossman wrote: post #516
mabbo wrote: post #66

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?

My website that runs on US-East-1 is up. However, my Alexa (Echo) won't control my thermostat right now. And my Ring app won't bring up my cameras. Those services are run on AWS.

Now I'm imagining someone dying because they couldn't turn their heating on because AWS. The 21st Century is fucked up.

Re: AWS us-east-1 outage

#552

Post by hoofhearted »

kingcharles wrote: post #531

Does McDonalds use AWS for the backend to their app? If I find out this is why I couldn't get my Happy Meal this morning I'm going to be really, really grumpy. EDIT: I'm REALLY grumpy now: https://aws.amazon.com/blogs/industries/aws-is-how-mcdonalds...

The McDonalds App was showing on the frontpage of Down Detector at the same time as all the Amazon dependent services last I checked.

Re: AWS us-east-1 outage

#553

Post by all_usernames »

25 Regions, 85 Availability Zones in this global cloud service and I can't login because of a failure in a single region (their oldest).

Can't login to AWS console at signin.aws.amazon.com:

   Unable to execute HTTP request: sts.us-east-1.amazonaws.com. Please try again.

Re: AWS us-east-1 outage

#554

Post by AtlasBarfed »

javajosh wrote: post #302
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…

That's a great philosophy.

Ok, let's take an organization, let's call them, say Ammizzun. Totally not Amazon. Let's say you have a very aggressive hire/fire policy which worked really well in rapid scaling and growth of your company. Now you have a million odd customers highly dependent on systems that were built by people that are now one? two? three? four? hire/fire generations up-or-out or cashed-out cycles ago.

So.... who owns it if the people that wrote it are lllloooooonnnnggg gone? Like, not just long gone one or two cycles ago so some institutional memory exists. I mean, GONE.

Re: AWS us-east-1 outage

#555

Post by alexatalktome »

dalrympm wrote: post #538

Contrary to what the status page says, CodePipeline is not working. Hitting the CLI I can start pipelines but they never complete and I get a lot of: Connection was closed before we received a valid response from endpoint URL: " https://codepipeline.us-east-1.amazonaws.com/ ".

Rumor is that our internal pipelines are the root cause. The CICD pipelines (not tests, the literal pipeline infrastructure) failed to block certain commits and pushed them to production when not ready.

We’ve been told to manually disable them to ensure integrity of our services when it recovers

Re: AWS us-east-1 outage

#556

Post by Crespyl »

kingcharles wrote: post #531

Does McDonalds use AWS for the backend to their app? If I find out this is why I couldn't get my Happy Meal this morning I'm going to be really, really grumpy. EDIT: I'm REALLY grumpy now: https://aws.amazon.com/blogs/industries/aws-is-how-mcdonalds...

Apparently Taco Bell too, not being able to place an order and then also not being able to fall back to McDonalds was how I realized there was a larger outage :p

What am I supposed to do for lunch now? Go to the drive through and order like a normal person? /s

Grumble grumble

Re: AWS us-east-1 outage

#557

Post by [deleted] »

tssva wrote: post #362
shepherdjerred wrote: post #143

Earlier quoted context omitted.

Does this imply Virginia is Godless?

Virginia's actual motto is "Sic semper tyrannis". What's more tyrannical than an omnipotent being that will condemn you to eternal torment if you don't worship them and follow their laws.

[deleted]

Re: AWS us-east-1 outage

#558

Post by jsperson »

kator wrote: post #328
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

I've worked for Amazon for 4 years, including stints at AWS, and even in my current role my team is involved in LSE's. I've never seen this behavior, the general culture has been find the problem, fix it, and then do root cause analysis to avoid it again. Jeff himself has said many times in All Hands and in public "Amazon is the best place to fail". Mainly because things will break, it's not that they break that's in…

I guess the question is why can't you (AWS) fix the problem of the status page not reflecting an outage? Maybe acceptable if the console has a hiccup, but when www.amazon.com isn't working right, there should be some yellow and red dots out there.

With the size of your customer base there were man years spent confirming the outage after checking the status.

Re: AWS us-east-1 outage

#559

Post by kingcharles »

saisundar wrote: post #474

Yikes, ring, the security system is also down. Wonder if crime rates might eventually spike up if aws goes down, in an utopian world where Amazon gets everyone to use ring.

I'm now imagining a team of criminals sitting around in face masks and hoodies refreshing AWS status page all day...

"AWS is down! Christmas came early boys! Roll out..."

Re: AWS us-east-1 outage

#560

Post by all_usernames »

jesboat wrote: post #484
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

Every damn Well-Architected Framework includes multi-AZ if not multi-region redundancy, and yet the single access point for their millions of customers is single-region. Facepalm in the form of $100Ms in service credits.
Post reply on HN