Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

301–310 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#301

Post by tanilama »

one2know wrote: post #182
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

COE won't lead to inevitable LE, it only means your manager wants you to be the scapegoat or you are indeed responsible for it.

Resolving COE can be a positive even if you know how to spin it, at least that was the case when I was there. But not sure whether things had changed

Re: AWS Cognito is having issues and health dashboards are still green

#302

Post by ipsocannibal »

ta20200710 wrote: post #272
uji wrote: post #195

Earlier quoted context omitted.

Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashboard or not. Having 'red' dashboard catches lot of eyes, so people responsible for making this decision always look at it from political point of view. As a dev oncall, we used to get 20 sev2s per day (an oncall ticket which needs to be handled within 15 mins)…

The ec2 or s3 dashboards showing red literally requires approval from ajassy himself irrc The status page is entirely manually updated.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

Re: AWS Cognito is having issues and health dashboards are still green

#303

Post by dodobirdlord »

one2know wrote: post #239
Bombthecat wrote: post #219

Earlier quoted context omitted.

I think the deeper problem is the interconnectivity between services and their apis. It's too complicated to maintain...

Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other manager…

Conveniently this gets tested during every new region launch. Each service is brought online in a sequence, and each service can only use other services running in the same region, which guarantees that no two services can be mutual startup dependencies. Sometimes region build-outs have to be paused when a circular dependency is discovered that has been introduced since the last region launch!

Re: AWS Cognito is having issues and health dashboards are still green

#304

Post by dodobirdlord »

WrtCdEvrydy wrote: post #113
riyadparvez wrote: post #109

Earlier quoted context omitted.

I've read this multiple times that AWS us-east-1 region is the one that has the highest number of outages. I am eager to hear others' experiences here.

us-east-1 is the zone with highest load and most new services are tested there first. rumor has it, some of the older hardware is moved there and that's why prices are a little cheaper but I have not been able to confirm that.

It’s not that the oldest hardware is moved there, it’s just that the oldest hardware was there to begin with. There are probably still first-generation EC2 instances running in us-east-1 on their original platforms.

Re: AWS Cognito is having issues and health dashboards are still green

#306

Post by o-__-o »

dilyevsky wrote: post #293
etaioinshrdlu wrote: post #247

Earlier quoted context omitted.

Wow. If I were in charge, the team running a service should not be the same team who decides whether a given service is healthy. This is pretty damaging info about the unprofessional way AWS actually appears to be run.

Guess what - most cloud providers are like that. My personal experience is with GCP where stuff can be majorly on fire and no status update for hours. Cloud SLOs are lies like a lot of other things there

My company will update their status but puts the most vague responses up. Reason is because we don’t want to appear inept when we crash the website. For example, because we ran out of disk space

Our competitors would have a field day with that

Re: AWS Cognito is having issues and health dashboards are still green

#307

Post by beoberha »

Twirrim wrote: post #61
s_dev wrote: post #25

Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing. It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening. How did AWS status page compare with status.io/aws?

> Can anyone explain why status pages are so difficult. What is an outage? When does an outage reach sufficient scale that updating the status page is the right thing to do? I used to work for AWS, and now work for another cloud provider. One thing that's hard to communicate is the sheer scale that these services operate at, what that means architecturally, and how they tend to break. Outages, even just slight degrad…

Exactly this. I work for a cloud provider and there has been a ton of push in the last year or so to develop customer communication teams and involve them at the first inkling of an outage. We can identify the subset of customers affected and contact them directly. Just publically saying there’s an outage would cause much more chaos.

Re: AWS Cognito is having issues and health dashboards are still green

#309

Post by dodobirdlord »

kristianpaul wrote: post #173

CloudWatch is definitely one of those "AWS Primitives" services that side effects others when having problems, something similar happened with DynamoDB some years ago.

In early 2019 CloudWatch had a major outage that was particularly nasty, where instead of just outright failing to report metrics it reported a small percentage of metrics. As a result a lot of autoscaling groups and DynamoDB tables that were theoretically supposed to avoid scaling in during a metric outage still scaled in, because they saw it as a 90+% traffic reduction rather than a metric outage.

Re: AWS Cognito is having issues and health dashboards are still green

#310

Post by o-__-o »

totaldude87 wrote: post #280
uji wrote: post #195

Earlier quoted context omitted.

Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashboard or not. Having 'red' dashboard catches lot of eyes, so people responsible for making this decision always look at it from political point of view. As a dev oncall, we used to get 20 sev2s per day (an oncall ticket which needs to be handled within 15 mins)…

wow, so much to their "leadership principles" , the first one being "customer obsession" and "earning trust", from what I see, this doesn't accomplish either :|

I’ve got another good FAANG principal joke:

“Don’t be evil”

buys doubleclick

Post reply on HN