Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

41–50 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#41
post #22

There's a lot more going on over there... - 7 cloudfront distributions created today are still in "InProgress", a few already for more than one hour - The support case I created about it doesn't show up in my support portal. Direct link to it does work though

yeah, I’m seeing event bridge errors and am unable to load cloudwatch log groups. happy short staff day!

Re: AWS Cognito is having issues and health dashboards are still green

#44
post #35

"This issue has also affected our ability to post updates to the Service Health Dashboard." Last sentence of the alert at the top of the page.

Always seems to be the case -- this happened before where the status pages updates were stored in ... S3. It goes beyond coincidence when this happens several times in a row.

I think the other explainations sound plausible. There is no technical difficulty here that AWS can't solve -- it's political. Having an outage with a status page makes you liable for your SLAs.

Re: AWS Cognito is having issues and health dashboards are still green

#45
post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

It says on this outage page (as of 11:11 ET) that the problem with Kinesis is also causing problems updating this outage dashboard which may explain the delay?

Re: AWS Cognito is having issues and health dashboards are still green

#46

> This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces. Well, this is a major outgage

Indeed, we had the first AWS Kinesis issues already at 13:50 (UTC). Now it's still ongoing after two hours. The status page didn't even update in the first 45 min or so...

Re: AWS Cognito is having issues and health dashboards are still green

#47
post #33
post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

It is kind of perplexing that AWS dogfoods its own status page. I remember during the massive S3 outage a few years ago that their status page remained green almost the entire time because the red/green/blue icons for the status was stored in... wait for it... S3. You'd think they would have learned from that.

They did. It came up in the post incident report, and senior leadership kicked off work to have it run on its own distinct infrastructure so that this wouldn't happen again.

If you look at where the content on https://status.aws.amazon.com/ is actually hosted from you'll see things like the status icons are all hosted under the same domain, e.g. https://status.aws.amazon.com/images/status1.gif https://status.aws.amazon.com/images/status0.gif etc.

If you look at the source code for the site, you'll again see that everything is hosted from the same domain.

One of their main goals was to ensure that it could never go wrong that way again.

Re: AWS Cognito is having issues and health dashboards are still green

#49
post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

When your company gets sufficiently large, outages become political. Failure happens at the speed of computing but agreeing that something is failing in a way that customers need to be told about is a slower process. Even when status pages are fully automatic (rather than manually updated), there will tend to be gaming of the metrics that constitute that. Ideally you would just be monitoring your SLOs and publishing…

[deleted]
Post reply on HN