Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Cognito is having issues and health dashboards are still green

status.aws.amazon.com

Post reply on HN
Print view View original

311–320 of 369 posts

Re: AWS Cognito is having issues and health dashboards are still green

#311

Post by reaperducer »

ipsocannibal wrote: post #302
ta20200710 wrote: post #272

Earlier quoted context omitted.

The ec2 or s3 dashboards showing red literally requires approval from ajassy himself irrc The status page is entirely manually updated.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

All I see in your response is, "We don't want to tell the truth because it might cost us money."

Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.

Re: AWS Cognito is having issues and health dashboards are still green

#312

Post by [deleted] »

reaperducer wrote: post #311
ipsocannibal wrote: post #302

Earlier quoted context omitted.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

All I see in your response is, "We don't want to tell the truth because it might cost us money." Maybe if it started costing the company actual money, it might make the investments necessary to ensure it doesn't go down in the first place.

[deleted]

Re: AWS Cognito is having issues and health dashboards are still green

#313

Post by TheHydroImpulse »

one2know wrote: post #182
PragmaticPulp wrote: post #172

We hired an engineer out of Amazon AWS at a previous company. Whenever one of our cloud services went down, he would go to great lengths to not update our status dashboard. When we finally forced him to update the status page, he would only change it to yellow and write vague updates about how service might be degraded for some customers. He flat out refused to ever admit that the cloud services were down. After some…

At amazon, admitting to a problem will guaranteed lead to having to open a COE, correction of error, which means meetings with executives, inevitable "least effective" rating, development plan, scapegoating, PIP, and firing.

If anyone notices a problem you'll likely need to write a COE, there's no way to get around that. Not updating the status page absolutely doesn't get you out of that task.

COE also doesn't lead to negative marks on anyone at AWS that I know of. It's a learning experience to know why it happened and action items so it doesn't happen again.

Re: AWS Cognito is having issues and health dashboards are still green

#314

Post by rhencke »

ipsocannibal wrote: post #302
ta20200710 wrote: post #272

Earlier quoted context omitted.

The ec2 or s3 dashboards showing red literally requires approval from ajassy himself irrc The status page is entirely manually updated.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

So... then what's the point of a status dashboard?

Re: AWS Cognito is having issues and health dashboards are still green

#315

Post by jmtulloss »

ipsocannibal wrote: post #302
ta20200710 wrote: post #272

Earlier quoted context omitted.

The ec2 or s3 dashboards showing red literally requires approval from ajassy himself irrc The status page is entirely manually updated.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

This might be an oversimplification.

With any customer that has SLAs written into their contracts, they're not just going off your status page. They most likely have a direct point of contact and exact reporting will be done in the postmortem.

The status page is for customers for which there aren't significant legal or business complications and exists to provide transparency. In my opinion you do want "random" people at your company to be able to update it in order to provide very stressed out customers with the best information you have.

As an industry we probably should recognize this more explicitly and have more standard status pages that are like "everything might be broken but we're not sure yet"

Re: AWS Cognito is having issues and health dashboards are still green

#316

Post by throw_m239339 »

rhencke wrote: post #314
ipsocannibal wrote: post #302

Earlier quoted context omitted.

Flipping anything to red entails significant legal and business complications. For starters you are basically admitting that customers deserve a refund for services not provided. Im not surprised that execs must be involved in that decision. You don't want random developer making a decision that could incur millions of dollars in potential loses when there are other strictly non-techincal factors to consider.

So... then what's the point of a status dashboard?

> So... then what's the point of a status dashboard?

Exactly. Apparently it's just a marketing tool if you believe parent comments...

Re: AWS Cognito is having issues and health dashboards are still green

#317

Post by astatine »

The way ahead should be an independent entity who audits systems and has responsibility to certify that the dashboard represents a true and accurate view of the actual status. Like is done so effectively with company financials.

Oh, wait! EY, PWC, and who can forget Arthur Andersen!

But, naturally, technology people can solve this better than anyone else, right?

Re: AWS Cognito is having issues and health dashboards are still green

#319

Post by [deleted] »

rexreed wrote: post #91
ti_ranger wrote: post #89

Earlier quoted context omitted.

> Yes but only if you initiate a claim and follow their steps. Check out these onerous terms: There's most likely a reason for this. Like, maybe in the past AWS customers have tried claiming for SLA credits for incidents that didn't impact them, in order to reduce their bill.

This is backwards thinking. Why require customers to file a claim for what are obvious outages? Instead, AWS should automatically apply credits to those accounts that have paid for guaranteed uptime without requiring this whole silly claims process. The mechanism can be really simple. If AWS themselves posts an outage to their status page and/or some third-party service posts an outage then credits are immediately ap…

[deleted]

Re: AWS Cognito is having issues and health dashboards are still green

#320

Post by jjoonathan »

narism wrote: post #282
ultimoo wrote: post #274

Earlier quoted context omitted.

Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?

Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law

Yep. Everyone ships the org chart.
Post reply on HN