Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

181–190 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#181

Post by hypervisorxxx »

PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

Or it could be a fluke. GCP went down in such a way the dashboard updates were not independent of the regions that went down so the dashboard was down because the region down it was reporting on was the region it was deployed on. I think that ended up being the root cause but yes another Sunday on call I didn't go to the gym and sat infront of my computer waiting for updates that never came. What's worse is when they say they will update at a certain time next and then no update is made.

Even if you don't know what to say still update saying that so the rest of us can report to our teams and make decisions about our own worklives and personal lives.

Re: AWS Cognito is having issues and health dashboards are still green

#182

Post by one2know »

PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

Re: AWS Cognito is having issues and health dashboards are still green

#183

Post by rhizome »

kevindong wrote: post #171
piewzko wrote: post #128

Now is probably a good time to plug some of the open source alternatives to vendor locked in identity solutions: - https://github.com/ory - https://github.com/dexidp/dex - https://github.com/authelia/authelia - https://github.com/keycloak/keycloak - https://www.gluu.org/ - https://github.com/accounts-js/accounts

I'd expect Amazon to be better able to maintain uptime than a self-hosted option at most (but not all) companies.

Amazon can't diversify their providers, though.

Regular Joes like us can use AWS, GCE, on premises, some non-reseller colocation provider, etc., and create failover duplicates, alternative deploy targets, or simply not ever have a complete outage due to the unlikelihood of all of these things failing at once.

Re: AWS Cognito is having issues and health dashboards are still green

#184

Post by x86_64Ubuntu »

revicon wrote: post #143

I’m trying to find a doc on running cognito using multiple zones and I can’t find much. Anyone have a multi-az cognito deployment running right now?

Zones or regions? I don't think multi-az will help you when the whole region kicks the bucket.

Re: AWS Cognito is having issues and health dashboards are still green

#185

Post by rhizome »

chrisan wrote: post #159
PKop wrote: post #150

Earlier quoted context omitted.

>for operators trying to resolve this issue It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page. If you're right, why have the status page then? It is useless by your definition yes?

Not to mention it doesn't take a technical person to update the status page. Its even more frustrating when you are aware of problems early on and start talking to support and THEY don't even know about problems yet. Maybe the thousands of people is what prevents status from being updated, everyone tries to hide their own faults internally even

Heck, doesn't Amazon have an AI/ML product? Make the status page reflect sentiment analysis of support conversations.

Re: AWS Cognito is having issues and health dashboards are still green

#186

Post by throwaway2048 »

Aperocky wrote: post #178
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

That's opposite of my experience at AWS. It's likely that the culture at AWS has changed over the past few years, it's also likely that there's a difference in culture between teams.

One has to be wary of the differences between what is said in places like the employee handbook and espoused as official policy, and what actually ends up happening.

AWS and amazon in general espouse all sorts of values relating to taking responsibility and owning problems.

Whats left unstated is that the management structure hammers you to the wall as soon as they find somebody to blame.

Re: AWS Cognito is having issues and health dashboards are still green

#187

Post by grinich »

piewzko wrote: post #128

Now is probably a good time to plug some of the open source alternatives to vendor locked in identity solutions: - https://github.com/ory - https://github.com/dexidp/dex - https://github.com/authelia/authelia - https://github.com/keycloak/keycloak - https://www.gluu.org/ - https://github.com/accounts-js/accounts

Shameless plug for WorkOS. (I'm the founder. Hope that's still ok on HN!)

We're like Stripe for SSO/SAML auth. Docs here: https://workos.com/docs

Here's our HN launch: https://news.ycombinator.com/item?id=22607402

Re: AWS Cognito is having issues and health dashboards are still green

#189

Post by felixgallo »

one2know wrote: post #182
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

I've caused and authored many COEs at Amazon, and additionally have been involved in maybe fifty for neighboring teams. I can't recall a time it had a negative career impact for anyone, much less any of the consequences you list.

Re: AWS Cognito is having issues and health dashboards are still green

#190

Post by myleshenderson »

drfritznunkie wrote: post #122
sk5t wrote: post #108

Earlier quoted context omitted.

Eh? Brokering amongst multiple trusts (and managing protocol transition) is almost the raison d'etre for lifting token issuance out of your app and into ADFS, Okta, Auth0, etc. Of course you'll have to deal with home realm discovery--really need to go in with open eyes on that one.

Yes, but cognito endpoints and pools ids are regional and globally unique, and there is no way that I know of to setup duplicate userpools in multiple regions and have requests served by either region. That means the customer IDP side would need to have two different SAML apps configured for each region...

This was shared with me today: https://medium.com/@nealrp/aws-cross-region-cognito-replicat...
Post reply on HN