Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

231–240 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#231

Post by paragraft »

myleshenderson wrote: post #190
drfritznunkie wrote: post #122

Earlier quoted context omitted.

Yes, but cognito endpoints and pools ids are regional and globally unique, and there is no way that I know of to setup duplicate userpools in multiple regions and have requests served by either region. That means the customer IDP side would need to have two different SAML apps configured for each region...

This was shared with me today: https://medium.com/@nealrp/aws-cross-region-cognito-replicat...

That design raises the question as to what happens to passwords. Do they get replicated in the global table in plaintext? Or are you still forced to do a global user password reset if you want to failover to another user pool?

Re: AWS Cognito is having issues and health dashboards are still green

#232

Post by bart_spoon »

rcardo11 wrote: post #40

> This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces. Well, this is a major outgage

I'm also seeing weirdness with Batch. Its working, but the dashboards aren't showing job statuses accurately and jobs aren't always terminating.

Re: AWS Cognito is having issues and health dashboards are still green

#233

Post by that_guy_iain »

driverdan wrote: post #210

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable. Before someone replies and says use a different AZ, that's not possible for everyone. If you use a 3rd party service that is hosted on us-east-1 you can't do anything about it. For example, many Heroku services are broken because of this.

This is what SLAs are for. Especially if you're using a 3rd party service.

Re: AWS Cognito is having issues and health dashboards are still green

#234

Post by one2know »

kdmytro wrote: post #222
MadVikingGod wrote: post #211

Earlier quoted context omitted.

Not at AWS, retail Amazon, but what I saw was COEs were either normal business process or PIP material depending on which org you worked for. And sometimes just the excuse to get you gone. Where I was about 99.9% of the COEs where just a lesson learned and new process to prevent it. There was one that was basically used as a tool to remove a VERY good engineer, that didn't mesh well with new leadership. A sister org,…

What is PIP?

Amazon fires between 5-15% of engineers per year. PIP is to get you to quit. Amazon hires a TON of entry level SDE 1 engineers to sacrifice at the altar of Bezos so more shitty employees get to stay. Lifespan of a SDE 1 whipping boy/girl at Amazon, as a result is 3-6 months.

Re: AWS Cognito is having issues and health dashboards are still green

#235

Post by andrewstuart2 »

rhizome wrote: post #183
kevindong wrote: post #171

Earlier quoted context omitted.

I'd expect Amazon to be better able to maintain uptime than a self-hosted option at most (but not all) companies.

Amazon can't diversify their providers, though. Regular Joes like us can use AWS, GCE, on premises, some non-reseller colocation provider, etc., and create failover duplicates, alternative deploy targets, or simply not ever have a complete outage due to the unlikelihood of all of these things failing at once.

They diversify, they just do it at a completely different layer than a cloud consumer.

Re: AWS Cognito is having issues and health dashboards are still green

#236

Post by unclekev »

outworlder wrote: post #192
chizhik-pyzhik wrote: post #138

Earlier quoted context omitted.

Updating the status dashboard is pretty low priority for operators trying to resolve this issue. It requires escalation up the management chain and careful wording.

> Updating the status dashboard is pretty low priority for operators trying to resolve this issue Which is why, during incident responses, there has to be people in charge of communication. Both internal and external communication, and some of this can be further delegated. That's a poor excuse. > It requires escalation up the management chain and careful wording Careful wording is more important for external stakeho…

> Incident communication should be factual and concise.

Could not agree more. It's immensely frustrating working with organisations that spend more time trying to cover up the cause of a outage to external stakeholders than actually fixing the root cause.

The same organisations tend try and blame individuals for outages.

I think both are a symptom of businesses that embrace the "blame culture"

Re: AWS Cognito is having issues and health dashboards are still green

#237

Post by PKop »

mypalmike wrote: post #220
PKop wrote: post #150

Earlier quoted context omitted.

>for operators trying to resolve this issue It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page. If you're right, why have the status page then? It is useless by your definition yes?

Literally 99.9% of the employees have no more knowledge than you about the inner workings of a given AWS service. This isn't to forgive their lack of updating the status page, but large engineering orgs are never the knowledge monoliths you might imagine they are.

This isn't a question of knowing a service is down. We're assuming the team that is fixing the service, knows it is down. It was a question of not having the resources to direct literally any other person in the org to log into an admin panel and flip a toggle from green to red.

Re: AWS Cognito is having issues and health dashboards are still green

#238

Post by PKop »

Spivak wrote: post #160
PKop wrote: post #150

Earlier quoted context omitted.

>for operators trying to resolve this issue It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page. If you're right, why have the status page then? It is useless by your definition yes?

Just because it has a lag from “issues reported” to “confirmed outage” doesn’t mean it’s useless. Non-green means there are issues and Amazon is aware of them.

My comment was in the context of the assertion that a team that knows the service is down and is fixing it is too busy to update the status, therefore no one else can update this status. Certainly it is understandable that if the issue is unknown that status cannot be updated.

Re: AWS Cognito is having issues and health dashboards are still green

#239

Post by one2know »

Bombthecat wrote: post #219
driverdan wrote: post #210

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable. Before someone replies and says use a different AZ, that's not possible for everyone. If you use a 3rd party service that is hosted on us-east-1 you can't do anything about it. For example, many Heroku services are broken because of this.

I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...

Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other managers and VP's in the company. Unfortunately it leads to circular dependencies.

Re: AWS Cognito is having issues and health dashboards are still green

#240

Post by mdaniel »

odiroot wrote: post #212
rcardo11 wrote: post #40

> This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces. Well, this is a major outgage

It's always a DNS issue.

That's a tough one -- I'm usually with you that it's always either DNS or cert expiry, but my go-to "it's always ..." when discussing AWS is: it's always security groups

Heh, maybe they accidentally locked themselves out of IAM, since those are great fun to troubleshoot, also

Post reply on HN