Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

71–80 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#71

Post by mcintyre1994 »

unilynx wrote: post #59
LennyWhiteJr wrote: post #48

Earlier quoted context omitted.

Just us-east-1.

But some global services run through us-east-1 - eg Cloudfront is now broken too. So this is also affecting users who don't actually run anything in us-east-1 explicitly (or in the US at all)

I'm not seeing any issues here yet with S3 images/website buckets stored in eu-west-1 and served by CloudFront.

You're right that there's definitely some internal coupling though:

> If you want to require HTTPS between viewers and CloudFront, you must change the AWS Region to US East (N. Virginia) in the AWS Certificate Manager console before you request or import a certificate.

From https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

Re: AWS Cognito is having issues and health dashboards are still green

#73

Post by zucked »

turdnagel wrote: post #18

Isn't it common practice to host your status board on someone else's infrastructure? In 2017 there was an S3 issue that supposedly affected their ability to post. I believe they said that they were updating how they posted to the status board so that there would no longer be a dependency on S3. Well, I guess whatever they're dependent on now broke.

That day was a nightmare for a lot of people - it wasn't just S3 that went down, it was like all of US-EAST.

Luckily my company decided against multi-az for the cost savings so I spent all day firefighting.

Re: AWS Cognito is having issues and health dashboards are still green

#74

Post by rexreed »

ceejayoz wrote: post #56
rexreed wrote: post #27

Earlier quoted context omitted.

Exactly. Try telling your boss when you paid a premium for uptime available that you just wasted a whole lot of money because you aren't getting the uptime you paid for. If AWS can't actually guarantee uptime (maybe no one can), then they need to have in their terms an automatic credit on a per-minute basis for uptime that is not delivered but otherwise paid for in a "guarantee".

They do exactly that already. https://aws.amazon.com/legal/service-level-agreements/

Yes but only if you initiate a claim and follow their steps. Check out these onerous terms:

Credit Request and Payment Procedures

To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:

1. the words “SLA Credit Request” in the subject line;

2. the dates, times, and affected AWS region of each Unavailability incident that you are claiming;

3. the resource IDs for the affected Included Service ; and

4. your request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks).

If the Monthly Uptime Percentage of such request is confirmed by us and is less than the Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which your request is confirmed by us. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit.

Re: AWS Cognito is having issues and health dashboards are still green

#75

Post by sleepybrett »

Twirrim wrote: post #47
opmac wrote: post #33

Earlier quoted context omitted.

It is kind of perplexing that AWS dogfoods its own status page. I remember during the massive S3 outage a few years ago that their status page remained green almost the entire time because the red/green/blue icons for the status was stored in... wait for it... S3. You'd think they would have learned from that.

They did. It came up in the post incident report, and senior leadership kicked off work to have it run on its own distinct infrastructure so that this wouldn't happen again. If you look at where the content on https://status.aws.amazon.com/ is actually hosted from you'll see things like the status icons are all hosted under the same domain, e.g. https://status.aws.amazon.com/images/status1.gif https://status.aws.amaz…

Except they posted this: 7:30 AM PST: We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in error rates for Kinesis in the US-EAST-1 Region. It's not post on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD.

(SHD being the Service Health Dashboard)

Re: AWS Cognito is having issues and health dashboards are still green

#77

Post by acdha »

darkcha0s wrote: post #65
s_dev wrote: post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

I completely agree, but can we talk for a second how absurd it is charging 90$ for essentially a service that just pings your infrastructure?

Try undercutting it. At some point you’ll learn that the problem isn’t that simple, operations is a key part of the product and isn’t free, and people expect support for important services.

Re: AWS Cognito is having issues and health dashboards are still green

#78

Post by unilynx »

mcintyre1994 wrote: post #71
unilynx wrote: post #59

Earlier quoted context omitted.

But some global services run through us-east-1 - eg Cloudfront is now broken too. So this is also affecting users who don't actually run anything in us-east-1 explicitly (or in the US at all)

I'm not seeing any issues here yet with S3 images/website buckets stored in eu-west-1 and served by CloudFront. You're right that there's definitely some internal coupling though: > If you want to require HTTPS between viewers and CloudFront, you must change the AWS Region to US East (N. Virginia) in the AWS Certificate Manager console before you request or import a certificate. From https://docs.aws.amazon.com/Amazo…

Existing cloudfront is indeed fine. But creating or deleting distributions fails now.

(I think it's also pretty rare for an already configured cloudfront to suffer from issues on the control planes. Cloudfront configuration updates are painfully slow even under normal circumstances, and that's probably because the configuration is heavily replicated to all POPs)

Re: AWS Cognito is having issues and health dashboards are still green

#79

Post by booleanbetrayal »

rcardo11 wrote: post #40

> This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces. Well, this is a major outgage

This is also affecting Fargate (at least EKS) in that its scheduling system is broken. No way to get new pods.

Re: AWS Cognito is having issues and health dashboards are still green

#80

Post by bird_monster »

s_dev wrote: post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

They aren't difficult. Amazon has no interest in having a working status page. Amazon would prefer the appearance of always green checkmarks over actually having a status page.
Post reply on HN