Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

431–440 of 1001 posts

Re: AWS us-east-1 outage

#431

Post by jrochkind1 »

jedberg wrote: post #137
orangepurple wrote: post #120

Earlier quoted context omitted.

Being dishonest about SLAs seems to bear zero cost in this case?

It's not really dishonest though because there is nuance. Most everything in EC2 is still working it seems, just the console is down. So is it really down? It should probably be yellow but not red.

Heroku is currently having major problems. My stuff is still up, but I can't deploy any new versions. Heroku runs their stuff on AWS. I have heard reports of other companies who run on AWS also having degarded service and outages.

i'd say when other companies who run their infrastruture on AWS are going out, it's hard to argue it's not a real outage.

But AWS status _has_ changed to yellow at this point. Probably heroku could be completely down because of an AWS problem, and AWS status would still not show red. But at least yellow tells us there's a problem, the distinction between yellow and red probably only matters at this point to lawyers arguing about the AWS SLA, the rest of us know yellow means "problems", red will never be seen, and green means "maybe problems anyway".

I believe the entire us-east-1 could be entirely missing, and they'd still only put a yellow not a red on status page. After all, the other regions are all fine, right?

Re: AWS us-east-1 outage

#432

Post by devenvdev »

john37386 wrote: post #387

It seems a bit long to fix! They probably paint themselves in a corner just like facebook few weeks ago. This make me think; Could it be that one day the internet will have a total global outage and it will take few days to recover?

The only possible scenario I could come up with is someone crashing internet on purpose, e.g. some crazy uncatchable trojan that starts DDOSing everything. I doubt such scenario is feasible though...

Re: AWS us-east-1 outage

#433

Post by guenthert »

ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

Uh, four minutes to identify the root cause? Damn, those guys are on fire.

Re: AWS us-east-1 outage

#434

Post by alex_young »

EDIT: As pointed out below, I missed that this was for the Amazon Connect service, and not an update for all of US-EAST-1. Preserved for consistency, but obviously just a comprehension issue on my side.

At least the updates are amusing:

"9:18 AM PST We can confirm degraded Contact handling by agents in the US-EAST-1 Region. Agents may experience issues logging in or being connected with end-customers."

WTF is "contact handling", an "agent" or an "end-customer"?

How about something like "We are confirming that some users are not able to connect to AWS services in us-east-1. We're looking into it."

Re: AWS us-east-1 outage

#435

Post by mountainofdeath »

88913527 wrote: post #277
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

I don't think engineers can believe in no-blame analysis if they know it'll harm career growth. I can't unilaterally promote John Doe, I have to convince other leaders that John would do well the next level up. And in those discussions, they could bring up "but John has caused 3 incidents this year", and honestly, maybe they'd be right.

There is no such thing as "no-blame" analysis. Even in the best organizations with the best effort to avoid it, there is always a subconscious "this person did it". It doesn't help that these incidents serve as convenient places for others to leverage to climb their own career ladder at your expense.

Re: AWS us-east-1 outage

#436

Post by gbear0 »

zegl wrote: post #13

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

I assume each service has its own health check that checks the service is accessible from an internal location, thus most are green. However, when Service A requires Service B to do work, but Service B is down, a simple access check on Service A clearly doesn't give a good representation of uptime.

So what's a good health check actually report these days? Is it just about its own status, or should it include a breakdown of the status of external dependencies as part of its folded up status?

Re: AWS us-east-1 outage

#437

Post by dastbe »

alex_young wrote: post #434

EDIT: As pointed out below, I missed that this was for the Amazon Connect service, and not an update for all of US-EAST-1. Preserved for consistency, but obviously just a comprehension issue on my side. At least the updates are amusing: "9:18 AM PST We can confirm degraded Contact handling by agents in the US-EAST-1 Region. Agents may experience issues logging in or being connected with end-customers." WTF is "contac…

that’s an update for amazon connect, which is a customer support related service.

Re: AWS us-east-1 outage

#438

Post by jolux »

Kavelach wrote: post #381
ethbr0 wrote: post #235

Earlier quoted context omitted.

> It should be costing them trust not to push it when they should though. This is what Amazon, the startup, understood. Step 1: Always make it right and make the customer happy, even if it hurts in $. Step 2: If you find you're losing too much money over a particular issue, fix the issue . Amazon, one of the world's largest companies, seems to have forgotten that the risk of not reporting accurately isn't money, but…

> It's late Soviet Union in a nutshell How come an action of a private company in a capitalist country is like the Soviet Union?

Private companies are small centrally-planned economies within larger capitalist systems.

Re: AWS us-east-1 outage

#439

Post by detaro »

alex_young wrote: post #434

EDIT: As pointed out below, I missed that this was for the Amazon Connect service, and not an update for all of US-EAST-1. Preserved for consistency, but obviously just a comprehension issue on my side. At least the updates are amusing: "9:18 AM PST We can confirm degraded Contact handling by agents in the US-EAST-1 Region. Agents may experience issues logging in or being connected with end-customers." WTF is "contac…

Amazon Connect is a call-center product, so that report makes sense.

Re: AWS us-east-1 outage

#440

Post by dekhn »

dekhn wrote: post #146
jedberg wrote: post #137

Earlier quoted context omitted.

It's not really dishonest though because there is nuance. Most everything in EC2 is still working it seems, just the console is down. So is it really down? It should probably be yellow but not red.

if you cannot access the control plane to create or destroy resources, it is down (partial availability). The jobs that are running are basically zombies.

I'm right in the middle of an AWS-run training and we literally can't run the exercises because of this.

let me repeat that: my AWS trainign that is run by AWS that I pay AWS for isn't working, because AWS is having control plane (or other) issues. This is several hours after the initial incident. We're doing training in us-west-2, but the identity service and other components run in us-east-1.

Post reply on HN