AWS us-east-1 outage • Hacker News Board

Re: AWS us-east-1 outage

#151

Post by jolux » Tue, Dec 07, 2021, 4:21 PM UTC

jkingsman wrote: post #51

judge2020 wrote: post #34

Earlier quoted context omitted.

I don't see why they couldn't provide an error rate graph like Reddit[0] or simply make services yellow saying "increased error rate detected, investigating..." 0: https://www.redditstatus.com/#system-metrics

Because Amazon has $$$$$ in their SLOs, and it costs them through the nose every minute they're down in payments made to customers and fees refunded. I trust them and most companies not to be outright fraudulent (although I'm sure some are), but it's totally understandable they'd be reticent to push the "Downtime Alert/Cost Us a Ton of Money" button until they're sure something serious is happening.

It should be costing them trust not to push it when they should though. A trustworthy company will err on the side of pushing it. AWS is a near-monopoly, so their unprofessional business practices have still yet to cost them.

Re: AWS us-east-1 outage

#152

Post by jonhohle » Tue, Dec 07, 2021, 4:21 PM UTC

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

On the retail/marketplace side this wasn't my experience, but we also didn't have any public dashboards. On Prime we occasionally had to refund in bulk, and when it was called for (internally or externally) we would right up a detailed post-mortem. This wasn't fun, but it was never about blaming a person and more about finding flaws in process or monitoring.

Re: AWS us-east-1 outage

#153

Post by Bedon292 » Tue, Dec 07, 2021, 4:22 PM UTC

rickreynoldssf wrote: post #82

EC2 at least seems fine but the console is definitely busted as of 16:00UTC

I am still getting 'Invalid region parameter' for resources in us-east-1, the others are fine.

Re: AWS us-east-1 outage

#154

Post by sbr464 » Tue, Dec 07, 2021, 4:22 PM UTC

shepherdjerred wrote: post #143

sbr464 wrote: post #132

Earlier quoted context omitted.

Ohio's actual motto funnily kind of fits here: With God, all things are possible

Does this imply Virginia is Godless?

Maybe virginia is for lovers, ohio is for God/gods?

Re: AWS us-east-1 outage

#155

Post by grumple » Tue, Dec 07, 2021, 4:22 PM UTC

mabbo wrote: post #66

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?

I can tell you that some processes are not running, possibly due to SQS or SWF problems. Previous outages of this scale were caused by Kinesis outages. Can't connect via aws login at the cli either since we use SSO and that seems to be down.

Re: AWS us-east-1 outage

#156

Post by avsteele » Tue, Dec 07, 2021, 4:22 PM UTC

Getting strange errors trying to manage my amazon account right now, could this be related?

494 ERROR and "We're sorry Something went wrong with our website, please try again later."

Re: AWS us-east-1 outage

#157

Post by meepmorp » Tue, Dec 07, 2021, 4:22 PM UTC

mabbo wrote: post #66

Are the actual services down, or is it just the console and/or login page? For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start Are websites that run on AWS us-east up? Are the AWS CLIs working?

EventBridge, CloudWatch. I've just started getting session errors with the console, too.

Re: AWS us-east-1 outage

#158

Post by sirfz » Tue, Dec 07, 2021, 4:23 PM UTC

AWS ec2 API requests are failing and ECR also seems to be down (error 500 on pull)

Re: AWS us-east-1 outage

#159

Post by davecap1 » Tue, Dec 07, 2021, 4:23 PM UTC

meepmorp wrote: post #115

davecap1 wrote: post #20

Earlier quoted context omitted.

On top of that, the "Personalized Health Dashboard" doesn't work because I can't seem to log in to the console.

I'm logged in; you're missing an error message.

We have federated login with MFA required (which was failing). It just started working again.

Scratch that... console is not loading at all now :)

Re: AWS us-east-1 outage

#160

Post by m3nu » Tue, Dec 07, 2021, 4:23 PM UTC

crad wrote: post #65

We make heavy usage of Kinesis Firehose in us-east-1. Issues started ~1:24am ET and resolved around 7:31am ET. Then really kicked in at a much larger scale at 10:32am ET. We're now seeing failures with connections to RDS Postgres and other services. Console is completely unavailable to me.

Route53 is not updating new records. Console is also out.