Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

341–350 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#341

Post by ZeteticElench »

one2know wrote: post #239
Bombthecat wrote: post #219

Earlier quoted context omitted.

I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...

Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other manager…

I would think they have a dependency graph to avoid circular dependencies.

Re: AWS Cognito is having issues and health dashboards are still green

#342

Post by justin-8 »

Aperocky wrote: post #194
throwaway2048 wrote: post #186

Earlier quoted context omitted.

One has to be wary of the differences between what is said in places like the employee handbook and espoused as official policy, and what actually ends up happening. AWS and amazon in general espouse all sorts of values relating to taking responsibility and owning problems. Whats left unstated is that the management structure hammers you to the wall as soon as they find somebody to blame.

I'm speaking from experience, not handbooks or policies. In fact, AWS is the least 'blame game' playing company I've worked at. The mindset of fix the problem and not to find some scapegoat is strong at least in my org, I really do appreciate this because it aligns with my personal belief.

Same here, I've made some huge fuck ups in my time at Amazon, one of which I was pretty new and assumed I would be fired for; but one of the principal engineers on my team told me not to worry, these things happen and it's a blameless process where we're just trying to get to the root cause of the problem and ensure it never happens again; and it was exactly as he said. The CoE we presented said "An engineer from the xyz team..." and never mentioned a name once.

Re: AWS Cognito is having issues and health dashboards are still green

#343

Post by hyperdimension »

felixgallo wrote: post #189
one2know wrote: post #182

Earlier quoted context omitted.

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

I've caused and authored many COEs at Amazon, and additionally have been involved in maybe fifty for neighboring teams. I can't recall a time it had a negative career impact for anyone, much less any of the consequences you list.

>a negative career impact

Nothing against you personally of course, but I just have to congratulate whomever it was who came up with this gem of an euphemism. It's definitely going up there next to 'career-limiting move'.

Re: AWS Cognito is having issues and health dashboards are still green

#344

Post by woofgum »

owenmarshall wrote: post #285
LegitShady wrote: post #229

Earlier quoted context omitted.

While on its face PIP is a guide to getting someone to commit to a higher level of improvement, for many companies its a formal warning that you need to shape up or you're going to be let go.

> for many companies its a formal warning that you need to shape up or you're going to be let go. Patently incorrect. A PIP is management telling you that you need to seek alternative employment, now . Joking/sarcasm aside: I’ve never seen or heard someone who is placed on a PIP successfully “exit” the PIP. They exit the company or they’re exited from the company. PIPs seem to mark the start of the “we are building f…

> I’ve never seen or heard someone who is placed on a PIP successfully “exit” the PIP

I have and at Amazon and AWS. The pattern I have seen is medical related. Someone is on some sort of medication that is screwing with their abilities and don't realize it. I've seen multiple cases: one where it was meds that caused liver problems and the person didn't know they were supposed to get regularly testing (crappy doctor) and another where they found out meds they were on caused short-term memory loss. These surfaced during the PIPs and were fixed - and the folks got out fine.

Re: AWS Cognito is having issues and health dashboards are still green

#345

Post by one2know »

Jabberwock12 wrote: post #328
one2know wrote: post #300

Earlier quoted context omitted.

Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by thei…

I've been here for years in a senior capacity and I've never even heard of Simple Workflow. Some products, like Dynamo and Lambda became favorable internally to support the migration to native AWS. I'm not sure that assigning this to a perceived internal power grab aligns with reality. > obvious to me > would become very powerful > cabal

This was maybe five years ago. The overarching architecture meme is/was used to say "my technology XYZ is the basis behind these five director level orgs, therefore I should be made the VP of the XYZ project over these five areas."

Re: AWS Cognito is having issues and health dashboards are still green

#346

Post by one2know »

jschwartzi wrote: post #292
owenmarshall wrote: post #285

Earlier quoted context omitted.

> for many companies its a formal warning that you need to shape up or you're going to be let go. Patently incorrect. A PIP is management telling you that you need to seek alternative employment, now . Joking/sarcasm aside: I’ve never seen or heard someone who is placed on a PIP successfully “exit” the PIP. They exit the company or they’re exited from the company. PIPs seem to mark the start of the “we are building f…

I got PIP'ed and actually fixed the problem I had and resolved the PIP. The problem was that I would mis-ship items sometimes in a warehouse. I figured out that I couldn't reliably read some of the product labels, so I went to go get an eye exam. Apparently I had 20/100 vision in one eye due to astigmatism. Getting glasses meant that I quit fucking up, so they dropped the PIP and moved me into another part of the com…

Wow, I didn't know pickers and stowers got PIPs. Obviously you did a smart thing in that you went and got a medical diagnosis. The company would be facing a medical disability lawsuit if they followed through with the PIP/firing.

A lawyer I spoke with suggested employees regularly visit their doctor about work related stress so that when they inevitably get PIP'ed they can claim medical leave and work related illness. Some places its a war zone and that's what workers have to do.

Re: AWS Cognito is having issues and health dashboards are still green

#347

Post by avh02 »

that_guy_iain wrote: post #233
driverdan wrote: post #210

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable. Before someone replies and says use a different AZ, that's not possible for everyone. If you use a 3rd party service that is hosted on us-east-1 you can't do anything about it. For example, many Heroku services are broken because of this.

This is what SLAs are for. Especially if you're using a 3rd party service.

Yes, but those cute SLAs don't help your own customers who missed/lost/delayed things. Their downtime is your downtime, and with vendor lock in it means it's harder to just march elsewhere when there's a problem.

Re: AWS Cognito is having issues and health dashboards are still green

#348

Post by that_guy_iain »

avh02 wrote: post #347
that_guy_iain wrote: post #233

Earlier quoted context omitted.

This is what SLAs are for. Especially if you're using a 3rd party service.

Yes, but those cute SLAs don't help your own customers who missed/lost/delayed things. Their downtime is your downtime, and with vendor lock in it means it's harder to just march elsewhere when there's a problem.

They don't help your customers, but they help you recover the lost revenue. The penalties we have in our SLA result in our company doing things they really don't want to pay.

Re: AWS Cognito is having issues and health dashboards are still green

#349

Post by avh02 »

that_guy_iain wrote: post #348
avh02 wrote: post #347

Earlier quoted context omitted.

Yes, but those cute SLAs don't help your own customers who missed/lost/delayed things. Their downtime is your downtime, and with vendor lock in it means it's harder to just march elsewhere when there's a problem.

They don't help your customers, but they help you recover the lost revenue. The penalties we have in our SLA result in our company doing things they really don't want to pay.

fair, but if you lose your customer's trust that might not be recoverable. Depends on your business

Re: AWS Cognito is having issues and health dashboards are still green

#350

Post by Ensorceled »

geertj wrote: post #271
Ensorceled wrote: post #174

Earlier quoted context omitted.

"Shooting the messenger" is so common that we, well, have a phrase for it.

There is a phrase for it, but it does not match my experience at AWS at all. (Source: been working there for 3.5 years now). Things break, we do COEs and we learn from them. If an issue was caused by operator error, the COE would look at what missing or broken processes caused the operator to be able to make this error in the first place.

And yet we continue to have “all greens” on the status boards during outages.
Post reply on HN