Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

291–300 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#291

Earlier quoted context omitted.

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

What the hell is a COE? I hate that nobody seems to bother defining their acronyms anymore.

The acronym was literally defined immediately after it was used! Come on!

Re: AWS Cognito is having issues and health dashboards are still green

#292

Earlier quoted context omitted.

While on its face PIP is a guide to getting someone to commit to a higher level of improvement, for many companies its a formal warning that you need to shape up or you're going to be let go.

> for many companies its a formal warning that you need to shape up or you're going to be let go. Patently incorrect. A PIP is management telling you that you need to seek alternative employment, now . Joking/sarcasm aside: I’ve never seen or heard someone who is placed on a PIP successfully “exit” the PIP. They exit the company or they’re exited from the company. PIPs seem to mark the start of the “we are building f…

I got PIP'ed and actually fixed the problem I had and resolved the PIP. The problem was that I would mis-ship items sometimes in a warehouse. I figured out that I couldn't reliably read some of the product labels, so I went to go get an eye exam. Apparently I had 20/100 vision in one eye due to astigmatism. Getting glasses meant that I quit fucking up, so they dropped the PIP and moved me into another part of the company.

I guess I'm the poster child for having vision insurance as a company benefit.

Re: AWS Cognito is having issues and health dashboards are still green

#293
post #195

Earlier quoted context omitted.

Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashboard or not. Having 'red' dashboard catches lot of eyes, so people responsible for making this decision always look at it from political point of view. As a dev oncall, we used to get 20 sev2s per day (an oncall ticket which needs to be handled within 15 mins)…

Wow. If I were in charge, the team running a service should not be the same team who decides whether a given service is healthy. This is pretty damaging info about the unprofessional way AWS actually appears to be run.

Guess what - most cloud providers are like that. My personal experience is with GCP where stuff can be majorly on fire and no status update for hours. Cloud SLOs are lies like a lot of other things there

Re: AWS Cognito is having issues and health dashboards are still green

#294
post #221

Earlier quoted context omitted.

Yeah, had the same experience at a previous company. It's very frustrating that your transparency gets used against you by unscrupulous competitors.

How is it unscrupulous? This sort of shit happens all the time at all levels. Companies use each other’s public specs in their competition all the time. Or capitalizing on features like headphone jacks etc. in their ads before proceeding to remove them from their own products anyway (Samsung and Google) and so on.

[deleted]

Re: AWS Cognito is having issues and health dashboards are still green

#295
post #19

what a scam. who can hold them accountable for cheating those who paid for uptime guarantees? I guess the lawyers of those who paid for uptime guarantees...

wah wah wah, my boss is chewing my ear out cuz I can't explain to him that it's a vendor (AWS) outage host on-premise then, and deal with power outages, and infrastructure management, and failing hardware shit happens, drink a coffee and sit back for 30 mins

You could, at rush hour, drive to microcenter, buy all the components, drive to home depot and buy a generator and gas can, go back to the office and assemble everything, fill and turn on the generator...in less time than this outage has lasted. I think generally more people are concerned about the scope and length of the outage, rather than that outages occur. And in this instance the fact that they aren't admitting to their downtime at all...

Re: AWS Cognito is having issues and health dashboards are still green

#296

Earlier quoted context omitted.

Eventually, anyone in that role would get fired. No service has an established 100% availability uptime when measured over its complete existence (welcome to any assertions challenging this, if anyone has any).

Bitcoin?

Forks?

Re: AWS Cognito is having issues and health dashboards are still green

#297
post #290

Earlier quoted context omitted.

Performance Improvement Plan, they are not unique to Amazon, most places have them though the process may differ. Not to be too cynical but ultimately they’re a way to document that you’re not meeting expectations - before being fired. Should there be any sort of employment claim later its a mechanism by which an employer can show documentation that any issues related to your being let go were performance related and…

I think the reason they don’t work is because someone doesn’t just magically become a better employee over two months.

I think that's a bit simplistic. I've had coworkers that became better employees over time. The "problem" with PIPs is by the time you've screwed up long enough to be put on a PIP everyone knows there's no turning back.

For example, a friend I have that recently left Facebook knew for a good 6 months he needed to shape up. But they hadn't put him on a PIP in that time. They eventually offered him a decent severance to quit, and he took that rather than continuing to try. If he stayed, he probably would have been put on a PIP fairly shortly. It was the best thing for everyone. He wasn't all that happy there anyways.

Re: AWS Cognito is having issues and health dashboards are still green

#298
post #221

Earlier quoted context omitted.

Yeah, had the same experience at a previous company. It's very frustrating that your transparency gets used against you by unscrupulous competitors.

How is it unscrupulous? This sort of shit happens all the time at all levels. Companies use each other’s public specs in their competition all the time. Or capitalizing on features like headphone jacks etc. in their ads before proceeding to remove them from their own products anyway (Samsung and Google) and so on.

Just because it happens all the time doesn't mean it isn't unscrupulous.

Re: AWS Cognito is having issues and health dashboards are still green

#299

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

That's opposite of my experience at AWS. It's likely that the culture at AWS has changed over the past few years, it's also likely that there's a difference in culture between teams.

As a customer, it does seem consistent that the status dashboard doesn't say a service is down until it has been down for quite a while.

Re: AWS Cognito is having issues and health dashboards are still green

#300

Earlier quoted context omitted.

Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other manager…

Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?

Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful.

In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API.

Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.

Post reply on HN