Earlier quoted context omitted.
> The problem is that AWS makes you into an owner and that includes owning your career. Firing me for correctly telling customers that their services are down is not my idea of making me an owner.
Doing what you think is right, is not necessarily the right thing to do. This is why there is also "Disagree and Commit". There are many facets to this and I am 100% sure that you did not get fired for >correctly< telling customers... You could potentially get fired for incorrectly telling them though, if the issue was severe enough.
AWS Cognito is having issues and health dashboards are still green
351–360 of 369 posts
Re: AWS Cognito is having issues and health dashboards are still green
#352Earlier quoted context omitted.
I find it surprising how many people forget how much underlying business motives drive pretty much every action they make and how this is quickly forgotten by many. No matter how much you value science and engineering, it ultimately doesn't matter to the business unless that aligns directly with their revenue stream. Sometimes it does, sometimes it doesn't.
Yes. But I wonder if self-hosting their status page is really the correct decision from a marketing perspective. The people who consumes the status page on say Google Cloud probably know that Google self-hosting it is a bad decision from a technical point of view. So to the only people who care, their choice appear stupid. So I don't really understand what they gain by doing it. I think maybe I am wrong about it bein…
Few companies really respect their engineering teams/divisions in any sensible form from my experience, though I'm biased (even in heavy R&D environments). You're simply a means to an ends.
I understand your point though (and identify with it), but I find any mechanism/option that provides a way of containing potentially damaging information is going to be pushed by management over the option to release damaging information that a responsible engineer may want to disclose.
You're in a culture where admitting fault or liability is like pulling teeth and ripping finger nails off. It shouldn't be IMHO (we should own up to our mistakes and be reasonably forgiven), but that's unfortunately not the culture we have.
Re: AWS Cognito is having issues and health dashboards are still green
#353Now is probably a good time to plug some of the open source alternatives to vendor locked in identity solutions: - https://github.com/ory - https://github.com/dexidp/dex - https://github.com/authelia/authelia - https://github.com/keycloak/keycloak - https://www.gluu.org/ - https://github.com/accounts-js/accounts
Disclosure: I'm an employee of FusionAuth, and while there is a forever free community edition, it is free as in beer, not as in speech.
Re: AWS Cognito is having issues and health dashboards are still green
#354Earlier quoted context omitted.
It's funny that you point to that as the problem. The problem is more AWS' toxic engineering culture that has engineers fearing for their jobs in a way that guides their decision making. It's bad company culture, end of story.
AWS is big. Amazon is even bigger. Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general. You don't hear a lot of people praising AWS, the same way you don't hear a lot of people saying how great it is to have an iPhone. If I am happy, I have little incentive to post about it, since that should be the default state. But t…
Disgruntled people are the ones who often cry the loudest. Just because there may be teams who act like this, doesn't mean that is the case in general.
Is right up there with "we don't know it wasn't aliens"Re: AWS Cognito is having issues and health dashboards are still green
#355Earlier quoted context omitted.
Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.
All I see in your response is, "We don't want to tell the truth because it might cost us money." Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.
You have all the power you need to make the company change its behavior. Vote with your dollar and move to a different platform. I'm sure you have recommendations to share.
Re: AWS Cognito is having issues and health dashboards are still green
#356Earlier quoted context omitted.
Oh, what a pipedream. If only capitalism worked how it was described in textbooks. It turns out there are much easier lower cost optimizations businesses can perform based on managing perception rather than worrying about pesky concepts like utility.
You raise an interesting point. Where I work, most of our public status dashboards update to yellow or red automatically, with only a few failure conditions requiring a manual update. It’s always made me wonder whether we’ll ever get around to implementing capitalism with some manual update only dashboards.
Re: AWS Cognito is having issues and health dashboards are still green
#357Large-scale events (LSEs) are becoming more and more common. It'll keep getting worse. AWS has to take a hard look at how they build their software. Their bad engineering practices will eventually catch up to them. You can't treat AWS the same as Alexa. Sometimes it's smarter to take your time to ship stuff instead of putting it out there. Burning out your oncall engineers is not a feasible long-term plan. AWS will b…
You seem to have insight on AWS's engineering practices. From your point of view what should be changed?
Re: AWS Cognito is having issues and health dashboards are still green
#358Earlier quoted context omitted.
Thats too bad, I always imagined the backend was as magical as what AWS users see. I still wish I could have a peek at how S3 works or IAM. Not enough to get a job at AWS - I know they'd fire me the first time I left early for a parent teacher conference or took a sick day, so why put myself in that position.
The only thing magical about AWS' backend is how much manpower they can throw at things. Amazon doesn't have a good engineering culture. It's all about shipping things as fast as possible. People get promoted an leave for other teams, and the new folk gets burned out due to on-call load while trying to fix crappy software they have inherited.
Why don't the new folk iteratively refactor their systems to remove operational burden? Isn't that part of owning any codebase you didn't write?
Re: AWS Cognito is having issues and health dashboards are still green
#359We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…
No idea what happens on AWS as I don't work there, but I have another perspective on this. There are perverse incentives to NOT update your status dashboard. Once I was asked by management to _take our status dashboard down_ . That sounded backwards, so I dug a bit more. Turns out our competitor was using our status dashboard as ammo against us in their sales pitch. Their claim was that we had too many issues and wer…
Re: AWS Cognito is having issues and health dashboards are still green
#360Earlier quoted context omitted.
You raise an interesting point. Where I work, most of our public status dashboards update to yellow or red automatically, with only a few failure conditions requiring a manual update. It’s always made me wonder whether we’ll ever get around to implementing capitalism with some manual update only dashboards.
Given enough law suits and mistakes by dev flipping dashboards to red with a bad code change or network provider outage and your org will have a manual public facing dashboard as well.