Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

361–369 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#361

Post by jschwartzi »

iamtedd wrote: post #338
jschwartzi wrote: post #292

Earlier quoted context omitted.

I got PIP'ed and actually fixed the problem I had and resolved the PIP. The problem was that I would mis-ship items sometimes in a warehouse. I figured out that I couldn't reliably read some of the product labels, so I went to go get an eye exam. Apparently I had 20/100 vision in one eye due to astigmatism. Getting glasses meant that I quit fucking up, so they dropped the PIP and moved me into another part of the com…

Was it management's decision to move you or yours? If it was theirs', it seems like management didn't have confidence you could improve once the problem was found and fixed. Kinda like changing two things at a time when troubleshooting. How did you feel about that?

It was mine. An opening appeared in the service department and I applied for it.

Re: AWS Cognito is having issues and health dashboards are still green

#362

Post by jschwartzi »

one2know wrote: post #346
jschwartzi wrote: post #292

Earlier quoted context omitted.

I got PIP'ed and actually fixed the problem I had and resolved the PIP. The problem was that I would mis-ship items sometimes in a warehouse. I figured out that I couldn't reliably read some of the product labels, so I went to go get an eye exam. Apparently I had 20/100 vision in one eye due to astigmatism. Getting glasses meant that I quit fucking up, so they dropped the PIP and moved me into another part of the com…

Wow, I didn't know pickers and stowers got PIPs. Obviously you did a smart thing in that you went and got a medical diagnosis. The company would be facing a medical disability lawsuit if they followed through with the PIP/firing. A lawyer I spoke with suggested employees regularly visit their doctor about work related stress so that when they inevitably get PIP'ed they can claim medical leave and work related illness…

I was a warehouse clerk which meant I was responsible for picking stowing, receiving, shipping and organizing the warehouse.

Re: AWS Cognito is having issues and health dashboards are still green

#363

Post by jmartens »

latch wrote: post #54
s_dev wrote: post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

Status pages, like SLAs, are sales tool - not engineering tools. At best, they are there to help decision makers go through their checklist. At worse, they exist to deceive.

1 million percent!

Which makes me wonder, why do we all rely on status pages rather than solve the problem ourselves in ways that don't require us to rely on the vendor?

Re: AWS Cognito is having issues and health dashboards are still green

#364

Post by jmartens »

turdnagel wrote: post #18

Isn't it common practice to host your status board on someone else's infrastructure? In 2017 there was an S3 issue that supposedly affected their ability to post. I believe they said that they were updating how they posted to the status board so that there would no longer be a dependency on S3. Well, I guess whatever they're dependent on now broke.

In a world where we can do virtually anything we want with technology, why do we rely on vendors updating their own individual status pages?

Re: AWS Cognito is having issues and health dashboards are still green

#365

Post by jmartens »

PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

Blaming people/employees is bad. That said, the idea of not updating a status page quickly, to reflect reality, is a problem at almost every SaaS company in the world. As others have said, status page changes are political and impact marketing, they have very little to do with providing good, timely information to customers.

Re: AWS Cognito is having issues and health dashboards are still green

#366

Post by jmartens »

darkcha0s wrote: post #65
s_dev wrote: post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

I completely agree, but can we talk for a second how absurd it is charging 90$ for essentially a service that just pings your infrastructure?

Except, the option to ping a service in order to programmatically inform a status page is almost never used. The dirty secret of status pages is that they are almost always manually updated, typically only when a very high bar is met, and after senior managers, sometimes even comms people, approve it.

Re: AWS Cognito is having issues and health dashboards are still green

#367

Post by iamtedd »

jschwartzi wrote: post #361
iamtedd wrote: post #338

Earlier quoted context omitted.

Was it management's decision to move you or yours? If it was theirs', it seems like management didn't have confidence you could improve once the problem was found and fixed. Kinda like changing two things at a time when troubleshooting. How did you feel about that?

It was mine. An opening appeared in the service department and I applied for it.

Thanks for replying :)

Re: AWS Cognito is having issues and health dashboards are still green

#368

Post by glenngillen »

riyadparvez wrote: post #109
maletor wrote: post #81

Earlier quoted context omitted.

Isn't that just called us-east-1?

I've read this multiple times that AWS us-east-1 region is the one that has the highest number of outages. I am eager to hear others' experiences here.

People are just projecting their own cognitive biases.

As Werner has said before everything fails all the time, so you need to design your system/architecture to accept that constant. US-east-1 is by far the largest of the regions, and at that scale you can probably assume that at any given point in time there is hardware in there failing that needs to be physically replaced. As a result it's the region most well equipped to tolerate that level of constant failure (it's got 6 AZs!). It's also the the most popular of the regions, is typically one of the launch regions for new services, and runs a bunch of critical Amazon infra too. If anything it holds a special place in terms of importance for AWS to keep it up because the impact of a widespread problem here is amplified. For the same reason though any problem here is much more visible across the entire internet. Which is why the handful of outages are so memorable to people.

Re: AWS Cognito is having issues and health dashboards are still green

#369

Post by mooreds »

technics256 wrote: post #201
piewzko wrote: post #128

Now is probably a good time to plug some of the open source alternatives to vendor locked in identity solutions: - https://github.com/ory - https://github.com/dexidp/dex - https://github.com/authelia/authelia - https://github.com/keycloak/keycloak - https://www.gluu.org/ - https://github.com/accounts-js/accounts

Anyone have thoughts on their experience with keycloak?

I haven't used it, but heard it's ... complex to get set up and run. (Again, I work for a competitor.)

Here's a reddit with a bunch of posts you could sift through: https://www.reddit.com/r/KeyCloak/

Post reply on HN