Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

211–220 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#211

Post by MadVikingGod »

stevepotter wrote: post #193
felixgallo wrote: post #189

Earlier quoted context omitted.

I've caused and authored many COEs at Amazon, and additionally have been involved in maybe fifty for neighboring teams. I can't recall a time it had a negative career impact for anyone, much less any of the consequences you list.

I work at AWS now and can second that. Nobody is happy when things break, but COEs are looked a positively and are circulated constantly to prevent repeats.

Not at AWS, retail Amazon, but what I saw was COEs were either normal business process or PIP material depending on which org you worked for. And sometimes just the excuse to get you gone.

Where I was about 99.9% of the COEs where just a lesson learned and new process to prevent it. There was one that was basically used as a tool to remove a VERY good engineer, that didn't mesh well with new leadership.

A sister org, one I worked a lot with, wouldn't COE anything. If you were the lead engineer on a product or service that had a COE you were going to get a PIP by year end review. I wasn't surprised when all the talent left that group.

Re: AWS Cognito is having issues and health dashboards are still green

#212

Post by odiroot »

rcardo11 wrote: post #40

> This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces. Well, this is a major outgage

It's always a DNS issue.

Re: AWS Cognito is having issues and health dashboards are still green

#213

Post by Aperocky »

one2know wrote: post #182
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

Having authored COEs before, none of your sentence after 'which means' was true at least in my case, nor are they true for COEs that I know of.

Re: AWS Cognito is having issues and health dashboards are still green

#214

Post by kenhwang »

freehunter wrote: post #70
rexreed wrote: post #14

Earlier quoted context omitted.

This is a relevant comment. I agree. Who will hold them accountable for violating uptime guarantees? Nobody? Then what's the point or purpose of an uptime guarantee? Marketing value?

If it’s in a contract, companies can sue. And big enough customers who have lost enough money due to the outage would definitely threaten to sue to recover that money.

We just ask nicely. Never really had a problem getting a huge % discount on the bill because of an outage. Extra bonus for us since there's no bottom line impact since we can tolerate some downtime (just annoying for engineering).

Re: AWS Cognito is having issues and health dashboards are still green

#215

Post by throwaway343432 »

zxcvbn4038 wrote: post #169
salil999 wrote: post #119

Earlier quoted context omitted.

I work at AWS. I can tell you surely enough it's not pretty or easy to work with. Design and architecture are great here but implementation of that is pretty crap...

Thats too bad, I always imagined the backend was as magical as what AWS users see. I still wish I could have a peek at how S3 works or IAM. Not enough to get a job at AWS - I know they'd fire me the first time I left early for a parent teacher conference or took a sick day, so why put myself in that position.

The only thing magical about AWS' backend is how much manpower they can throw at things.

Amazon doesn't have a good engineering culture. It's all about shipping things as fast as possible. People get promoted an leave for other teams, and the new folk gets burned out due to on-call load while trying to fix crappy software they have inherited.

Re: AWS Cognito is having issues and health dashboards are still green

#216

Post by iameli »

Aperocky wrote: post #178
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

That's opposite of my experience at AWS. It's likely that the culture at AWS has changed over the past few years, it's also likely that there's a difference in culture between teams.

Having talked to dozens of Amazon engineers, the only consistent picture I've formed in my head is that the culture varies wildly between teams. The folks on the happiest teams are always aghast at hearing the horror stories.

Re: AWS Cognito is having issues and health dashboards are still green

#217

Post by throwaway343432 »

Large-scale events (LSEs) are becoming more and more common. It'll keep getting worse.

AWS has to take a hard look at how they build their software. Their bad engineering practices will eventually catch up to them. You can't treat AWS the same as Alexa. Sometimes it's smarter to take your time to ship stuff instead of putting it out there. Burning out your oncall engineers is not a feasible long-term plan.

AWS will be in deep trouble when/if GCE fixes their customer support.

Re: AWS Cognito is having issues and health dashboards are still green

#218

Post by [deleted] »

throwaway343432 wrote: post #215
zxcvbn4038 wrote: post #169

Earlier quoted context omitted.

Thats too bad, I always imagined the backend was as magical as what AWS users see. I still wish I could have a peek at how S3 works or IAM. Not enough to get a job at AWS - I know they'd fire me the first time I left early for a parent teacher conference or took a sick day, so why put myself in that position.

The only thing magical about AWS' backend is how much manpower they can throw at things. Amazon doesn't have a good engineering culture. It's all about shipping things as fast as possible. People get promoted an leave for other teams, and the new folk gets burned out due to on-call load while trying to fix crappy software they have inherited.

[deleted]

Re: AWS Cognito is having issues and health dashboards are still green

#219

Post by Bombthecat »

driverdan wrote: post #210

Five hours later and nothing has changed. For a company like Amazon this should be unacceptable. Before someone replies and says use a different AZ, that's not possible for everyone. If you use a 3rd party service that is hosted on us-east-1 you can't do anything about it. For example, many Heroku services are broken because of this.

I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...

Re: AWS Cognito is having issues and health dashboards are still green

#220

Post by mypalmike »

PKop wrote: post #150
chizhik-pyzhik wrote: post #138

Earlier quoted context omitted.

Updating the status dashboard is pretty low priority for operators trying to resolve this issue. It requires escalation up the management chain and careful wording.

>for operators trying to resolve this issue It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page. If you're right, why have the status page then? It is useless by your definition yes?

Literally 99.9% of the employees have no more knowledge than you about the inner workings of a given AWS service. This isn't to forgive their lack of updating the status page, but large engineering orgs are never the knowledge monoliths you might imagine they are.
Post reply on HN