Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

321–330 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#321

Post by Frost1x »

reaperducer wrote: post #311
ipsocannibal wrote: post #302

Earlier quoted context omitted.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

All I see in your response is, "We don't want to tell the truth because it might cost us money." Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.

Oh, what a pipedream. If only capitalism worked how it was described in textbooks. It turns out there are much easier lower cost optimizations businesses can perform based on managing perception rather than worrying about pesky concepts like utility.

Re: AWS Cognito is having issues and health dashboards are still green

#322

Post by chillfox »

jmtulloss wrote: post #315
ipsocannibal wrote: post #302

Earlier quoted context omitted.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

This might be an oversimplification. With any customer that has SLAs written into their contracts, they're not just going off your status page. They most likely have a direct point of contact and exact reporting will be done in the postmortem. The status page is for customers for which there aren't significant legal or business complications and exists to provide transparency. In my opinion you do want "random" peopl…

Status pages are generally so unreliable that we do our own monitoring of external cloud resources that we depend on.

Re: AWS Cognito is having issues and health dashboards are still green

#323

Post by Frost1x »

Aeolun wrote: post #289
martamorena9182 wrote: post #287

Earlier quoted context omitted.

AWS is big. Amazon is even bigger. Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general. You don't hear a lot of people praising AWS, the same way you don't hear a lot of people saying how great it is to have an iPhone. If I am happy, I have little incentive to post about it, since that should be the default state. But t…

> The problem is that AWS makes you into an owner and that includes owning your career. Firing me for correctly telling customers that their services are down is not my idea of making me an owner.

You're the owner of aspects like responsibility and risk but not the owner of aspects related to financial growth (I mean, your stock options are, but that's about it).

Re: AWS Cognito is having issues and health dashboards are still green

#324

Post by [deleted] »

totaldude87 wrote: post #280
uji wrote: post #195

Earlier quoted context omitted.

Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashboard or not. Having 'red' dashboard catches lot of eyes, so people responsible for making this decision always look at it from political point of view. As a dev oncall, we used to get 20 sev2s per day (an oncall ticket which needs to be handled within 15 mins)…

wow, so much to their "leadership principles" , the first one being "customer obsession" and "earning trust", from what I see, this doesn't accomplish either :|

[deleted]

Re: AWS Cognito is having issues and health dashboards are still green

#325

Post by woahAcademia »

Frost1x wrote: post #321
reaperducer wrote: post #311

Earlier quoted context omitted.

All I see in your response is, "We don't want to tell the truth because it might cost us money." Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.

Oh, what a pipedream. If only capitalism worked how it was described in textbooks. It turns out there are much easier lower cost optimizations businesses can perform based on managing perception rather than worrying about pesky concepts like utility.

I never considered Amazon capitalistic given their exploitation of the USPS.

I considered them this private company subsidized by taxes.

Re: AWS Cognito is having issues and health dashboards are still green

#326

Post by ah88 »

LegitShady wrote: post #229
smithza wrote: post #227

Earlier quoted context omitted.

PIP : Performance Improvement Plan I assume this to mean that it is an element of an individual's PIP, a formal process to set guides for getting someone to commit to a higher level of achievement.

While on its face PIP is a guide to getting someone to commit to a higher level of improvement, for many companies its a formal warning that you need to shape up or you're going to be let go.

Not Amazon but at an HR/leader meeting our HR disclosed 45% of people on PIPs end up staying with the company.

Re: AWS Cognito is having issues and health dashboards are still green

#327

Post by martamorena9182 »

Aeolun wrote: post #289
martamorena9182 wrote: post #287

Earlier quoted context omitted.

AWS is big. Amazon is even bigger. Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general. You don't hear a lot of people praising AWS, the same way you don't hear a lot of people saying how great it is to have an iPhone. If I am happy, I have little incentive to post about it, since that should be the default state. But t…

> The problem is that AWS makes you into an owner and that includes owning your career. Firing me for correctly telling customers that their services are down is not my idea of making me an owner.

Doing what you think is right, is not necessarily the right thing to do. This is why there is also "Disagree and Commit". There are many facets to this and I am 100% sure that you did not get fired for >correctly< telling customers... You could potentially get fired for incorrectly telling them though, if the issue was severe enough.

Re: AWS Cognito is having issues and health dashboards are still green

#328

Post by Jabberwock12 »

one2know wrote: post #300
ultimoo wrote: post #274

Earlier quoted context omitted.

Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?

Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by thei…

I've been here for years in a senior capacity and I've never even heard of Simple Workflow. Some products, like Dynamo and Lambda became favorable internally to support the migration to native AWS.

I'm not sure that assigning this to a perceived internal power grab aligns with reality.

> obvious to me

> would become very powerful

> cabal

Re: AWS Cognito is having issues and health dashboards are still green

#329

Post by jugg1es »

martamorena9182 wrote: post #287
alfalfasprout wrote: post #266

Earlier quoted context omitted.

It's funny that you point to that as the problem. The problem is more AWS' toxic engineering culture that has engineers fearing for their jobs in a way that guides their decision making. It's bad company culture, end of story.

AWS is big. Amazon is even bigger. Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general. You don't hear a lot of people praising AWS, the same way you don't hear a lot of people saying how great it is to have an iPhone. If I am happy, I have little incentive to post about it, since that should be the default state. But t…

"You don't hear a lot of people praising AWS"

You definitely hear a lot of people praising AWS.

Re: AWS Cognito is having issues and health dashboards are still green

#330

Post by Jabberwock12 »

felixgallo wrote: post #189
one2know wrote: post #182

Earlier quoted context omitted.

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

I've caused and authored many COEs at Amazon, and additionally have been involved in maybe fifty for neighboring teams. I can't recall a time it had a negative career impact for anyone, much less any of the consequences you list.

I too have caused and authored many COEs at Amazon. I have also been involved in 50 to 100 COEs written by other teams and have also observed no instances of this having a negative impact on anyones career. COEs are core to Amazon's learning experience and you can be assigned to write one for simply being unlucky enough to be oncall when the incident occurred.
Post reply on HN