AWS us-east-1 outage • Hacker News Board

Re: AWS us-east-1 outage

#171

Post by btilly » Tue, Dec 07, 2021, 4:25 PM UTC

KineticLensman wrote: post #134

AH4oFVbPT4f8 wrote: post #108

Earlier quoted context omitted.

amazon.com seems to be having problems too. I get something went wrong to a new design/layout which I assume is either new or a fail safe.

Looks okay right now to this UK user of amazon.co.uk

It depends on which Amazon region you are being served from.

It is very unlikely that Amazon would deliberately make your messages cross the Atlantic just to find an American region that is unable to serve you.

Re: AWS us-east-1 outage

#172

Post by [deleted] » Tue, Dec 07, 2021, 4:26 PM UTC

zegl wrote: post #13

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

[deleted]

Re: AWS us-east-1 outage

#173

Post by School-Cotton » Tue, Dec 07, 2021, 4:26 PM UTC

DoctorOW wrote: post #107

lordnacho wrote: post #56

This got me thinking, are there any major chat services that would go down if a particular AWS/GCP/etc data centre went down? You don't want your service to go down, plus your team's comms at the same time.

Slack is pretty much full AWS, I've been switched over to Teams so I can't check.

My company's Slack instance is currently fine.

Re: AWS us-east-1 outage

#174

Post by thayne » Tue, Dec 07, 2021, 4:26 PM UTC

whoknew1122 wrote: post #138

zegl wrote: post #13

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

The problem being that often times you can't actually update the status page. Most internal systems are down. We can't even update our product to say it's down, because accessing the product requires a process that is currently dead.

That's why your status page should be completely independent from the services it is monitoring (minus maybe something that automatically updates it). We use a third party to host our status page specifically so that we can update it even if all our systems are down.

Re: AWS us-east-1 outage

#175

Post by 2bitlobster » Tue, Dec 07, 2021, 4:26 PM UTC

mabbo wrote: post #66

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?

Interactive Video Service (IVS) is down too

Re: AWS us-east-1 outage

#176

Post by jedberg » Tue, Dec 07, 2021, 4:26 PM UTC

dekhn wrote: post #122

jedberg wrote: post #111

Earlier quoted context omitted.

From what I've heard it's mostly true. Not only the CEO but a few SVPs can approve it, but yes a human must approve the update and it must be a high level exec. Part of the reason is because their SLAs are based on that dashboard, and that dashboard going red has a financial cost to AWS, so like any financial cost, it needs approval.

Sure, but... that just raises more questions :) Taken literally what you are saying is the service could be down and an executive could override that, preventing them for paying customers for a service outage, even if the service did have an outage and the customer could prove it (screenshots, metrics from other cloud providers, many different folks see it). I'm sure there is some subtlety to this, but it does mean t…

Like I said I never worked there and this is all hearsay but there is a lot of nuance here being missed like partial outages.

Re: AWS us-east-1 outage

#177

Post by jacobkg » Tue, Dec 07, 2021, 4:27 PM UTC

AWS Connect is down, so our customer support phone system is down with it

Re: AWS us-east-1 outage

#178

Post by whalesalad » Tue, Dec 07, 2021, 4:27 PM UTC

I can still hit EC2 boxes and networking is okay. DynamoDB is 100% down for the count, every request is an Internal Server Error.

Re: AWS us-east-1 outage

#179

Post by anonu » Tue, Dec 07, 2021, 4:28 PM UTC

I think its just the console - as my EC2 in us-east-1 are still reachable.

Re: AWS us-east-1 outage

#180

Post by singlow » Tue, Dec 07, 2021, 4:28 PM UTC

valeness wrote: post #85

This is more than just east. I am seeing the same error on us-west-2 resources.

I am seeing some issues but only with services that have global aspects such as s3. I can't create an s3 bucket even though I am in us-west-2 because I think the names are globally unique and creating them depends on us-east-1.