Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

241–250 of 1001 posts

Re: AWS us-east-1 outage

#241

Post by jedberg »

dekhn wrote: post #146
jedberg wrote: post #137

Earlier quoted context omitted.

It's not really dishonest though because there is nuance. Most everything in EC2 is still working it seems, just the console is down. So is it really down? It should probably be yellow but not red.

if you cannot access the control plane to create or destroy resources, it is down (partial availability). The jobs that are running are basically zombies.

Seems like the API is still working and so is auto scaling. So they aren’t really zombies.

Partial availability isn’t the same as no availability.

Re: AWS us-east-1 outage

#242

Post by artificial »

maximedupre wrote: post #227
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

Or just take responsibility. People will respect you for doing that and you will demonstrate leadership.

Way more fun argument: Outages just, uh… uh… find a way.

Re: AWS us-east-1 outage

#243

Post by melony »

maximedupre wrote: post #227
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

Or just take responsibility. People will respect you for doing that and you will demonstrate leadership.

And the guy who doesn't take responsibility gets promoted. Employees are not responsible for failures of management to set a good culture.

Re: AWS us-east-1 outage

#244

Post by rozenmd »

mystcb wrote: post #213

Not sure it helps, but got this update from someone inside AWS a few moments ago. "We have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also services like S3. We are actively working towards recovery."

They finally updated the status page: https://status.aws.amazon.com/

Re: AWS us-east-1 outage

#245

Post by ipmb »

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/

> 8:22 AM PST We are investigating increased error rates for the AWS Management Console.

> 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/

Re: AWS us-east-1 outage

#247

Post by 1-6 »

A4ET8a8uTh0 wrote: post #220
1-6 wrote: post #201

Earlier quoted context omitted.

Perhaps reward structure should be changed to incentivize the post-mortems. There could be several flaws that run underreported otherwise. We may run into the problem of everything documented and possible deliberate acts but for a service that relies heavily on uptime, that’s a small price to pay for a bulletproof operation.

Then we would drown in a sea of meetings and 'lessons learned' emails. There is a reason for post-mortems, but there has to be balance.

I find post-mortems interesting to read through especially when it’s not my fault. Most of them would probably be routine to read through but there are occasional ones that make me cringe or laugh.

Post-mortems can sometime be thought of like safety training. There is a big imbalance of time dedicated to learning proper safety handling just for those small incidences.

Re: AWS us-east-1 outage

#248

Post by l72 »

whalesalad wrote: post #178

I can still hit EC2 boxes and networking is okay. DynamoDB is 100% down for the count, every request is an Internal Server Error.

We are also seeing lots of failures with DynamoDB across all our services in us-east-1.

Re: AWS us-east-1 outage

#249

Post by howdydoo »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

Manually updated status pages are an anti-pattern to begin with. At that point, why not just call it a blog?
Post reply on HN