Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

191–200 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#191

Post by judge2020 »

snazz wrote: post #11
_0o6v wrote: post #8

> It's not posted on SHD as the issue has impacted our ability to post there. Is that not a massive catch-22 for a service dashboard?

This has happened a few times before, actually. Dogfooding is good, but not for status pages! Cloudflare does it right for their status page ( https://www.cloudflarestatus.com ). They don't use Cloudflare itself for it (you can tell because /cdn-cgi/trace returns nothing), the actual backend is Atlassian Statuspage, their TLS certificate is issued by Let's Encrypt instead of Cloudflare itself, and it's on a completel…

They do use their own registrar though:

  $ whois cloudflarestatus.com
  Registrar: Cloudflare, Inc.

Re: AWS Cognito is having issues and health dashboards are still green

#192

Post by outworlder »

chizhik-pyzhik wrote: post #138
jjoonathan wrote: post #106

Earlier quoted context omitted.

That's typical. The AWS status page is a marketing gimmick whose job is to stay green, not a good faith attempt to assess and report status. If there's an outage, seeing it accurately reflected on the status page is the exception, not the rule.

Updating the status dashboard is pretty low priority for operators trying to resolve this issue. It requires escalation up the management chain and careful wording.

> Updating the status dashboard is pretty low priority for operators trying to resolve this issue

Which is why, during incident responses, there has to be people in charge of communication. Both internal and external communication, and some of this can be further delegated.

That's a poor excuse.

> It requires escalation up the management chain and careful wording

Careful wording is more important for external stakeholders who might not have the full context. If one is stepping in eggshells with internal management too, that's bad management. Incident communication should be factual and concise.

Re: AWS Cognito is having issues and health dashboards are still green

#193

Post by stevepotter »

felixgallo wrote: post #189
one2know wrote: post #182

Earlier quoted context omitted.

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

I've caused and authored many COEs at Amazon, and additionally have been involved in maybe fifty for neighboring teams. I can't recall a time it had a negative career impact for anyone, much less any of the consequences you list.

I work at AWS now and can second that. Nobody is happy when things break, but COEs are looked a positively and are circulated constantly to prevent repeats.

Re: AWS Cognito is having issues and health dashboards are still green

#194

Post by Aperocky »

throwaway2048 wrote: post #186
Aperocky wrote: post #178

Earlier quoted context omitted.

That's opposite of my experience at AWS. It's likely that the culture at AWS has changed over the past few years, it's also likely that there's a difference in culture between teams.

One has to be wary of the differences between what is said in places like the employee handbook and espoused as official policy, and what actually ends up happening. AWS and amazon in general espouse all sorts of values relating to taking responsibility and owning problems. Whats left unstated is that the management structure hammers you to the wall as soon as they find somebody to blame.

I'm speaking from experience, not handbooks or policies.

In fact, AWS is the least 'blame game' playing company I've worked at. The mindset of fix the problem and not to find some scapegoat is strong at least in my org, I really do appreciate this because it aligns with my personal belief.

Re: AWS Cognito is having issues and health dashboards are still green

#195

Post by uji »

PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

Have worked at AWS before, and I can attest to this. Whenever we had an outage, our director and senior manager would take a call on whether to update the dashboard or not.

Having 'red' dashboard catches lot of eyes, so people responsible for making this decision always look at it from political point of view.

As a dev oncall, we used to get 20 sev2s per day (an oncall ticket which needs to be handled within 15 mins) so most of the time things are broken, its just that its not visible to external customers through dashboard.

Re: AWS Cognito is having issues and health dashboards are still green

#196

Post by patch_cable »

throwaway2048 wrote: post #186
Aperocky wrote: post #178

Earlier quoted context omitted.

That's opposite of my experience at AWS. It's likely that the culture at AWS has changed over the past few years, it's also likely that there's a difference in culture between teams.

One has to be wary of the differences between what is said in places like the employee handbook and espoused as official policy, and what actually ends up happening. AWS and amazon in general espouse all sorts of values relating to taking responsibility and owning problems. Whats left unstated is that the management structure hammers you to the wall as soon as they find somebody to blame.

Echoing what Aperocky said, I worked for Amazon for about 10 years, across a number of different teams. Amazon has its share of problems, but assigning blame for outages was not one of them.

During my 10 years, I had multiple opportunities to break and then fix things. The breaking was always looked at as "these things happen" while the fixing was always commended.

Re: AWS Cognito is having issues and health dashboards are still green

#197

Post by that_guy_iain »

sheeshkebab wrote: post #155
salil999 wrote: post #119

Earlier quoted context omitted.

I work at AWS. I can tell you surely enough it's not pretty or easy to work with. Design and architecture are great here but implementation of that is pretty crap...

Why use it then? (api is crap, uptime is crap, limits are crap... politics?)

Because business authorised it's use. The final say on using AWS doesn't belong to tech but busines and AWS is very good at the sales game. I went to one of their conferences and it was mostly business people and sales pitches.

Re: AWS Cognito is having issues and health dashboards are still green

#198

Post by patch_cable »

felixgallo wrote: post #189
one2know wrote: post #182

Earlier quoted context omitted.

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

I've caused and authored many COEs at Amazon, and additionally have been involved in maybe fifty for neighboring teams. I can't recall a time it had a negative career impact for anyone, much less any of the consequences you list.

Same. If anything, a well written COE has had positive career impact.

Re: AWS Cognito is having issues and health dashboards are still green

#199

Post by uji »

one2know wrote: post #182
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

This is very true. It doesn't lead to PIP always but whole amazonian culture makes it difficult for the person to stay in team/company.

Writing COE is kind of admission of guilt and I have definitely seen promotions getting delayed. During perf-review, lot of times managers of other teams raise COE has a point against the person going for promotion.

Re: AWS Cognito is having issues and health dashboards are still green

#200

Post by orf »

zxcvbn4038 wrote: post #169
salil999 wrote: post #119

Earlier quoted context omitted.

I work at AWS. I can tell you surely enough it's not pretty or easy to work with. Design and architecture are great here but implementation of that is pretty crap...

Thats too bad, I always imagined the backend was as magical as what AWS users see. I still wish I could have a peek at how S3 works or IAM. Not enough to get a job at AWS - I know they'd fire me the first time I left early for a parent teacher conference or took a sick day, so why put myself in that position.

I would be beyond fascinated at how IAM works under the hood.
Post reply on HN