Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

351–360 of 1001 posts

Re: AWS us-east-1 outage

#351

Post by jacquesm »

nix23 wrote: post #310
88913527 wrote: post #277

Earlier quoted context omitted.

I don't think engineers can believe in no-blame analysis if they know it'll harm career growth. I can't unilaterally promote John Doe, I have to convince other leaders that John would do well the next level up. And in those discussions, they could bring up "but John has caused 3 incidents this year", and honestly, maybe they'd be right.

You know why SO-teams, firefighters and military pilots are so successful? -You don't hide anything -Errors will be made -After training/mission everyone talks about the errors (or potential ones) and how to prevent them -You don't make the same error twice Being afraid to make errors and learn from them creates a culture of hiding, a culture of denial and especially being afraid to take responsibility.

You can even make the same error twice but you better have much better explanation the second time around than you had the first time around because you already knew that what you did was risky and or failure prone.

But usually it isn't the same person making the same mistake, usually it is someone else making the same mistake and nobody thought of updating processes/documentation to the point that the error would have been caught in time. Maybe they'll fix that after the second time ;)

Re: AWS us-east-1 outage

#352

Post by 88913527 »

javajosh wrote: post #302
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…

Should I be penalized if an upstream dependency, owned by another team, fails? Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? These are real problems in the micro-services world, especially since I own UI and there are dozens of teams pumping out services, and I'm at the mercy of all of them. The best I can do is gracefully fail when services don't function in a healthy state.

Re: AWS us-east-1 outage

#354

Post by ChadyWady »

lordnacho wrote: post #56

This got me thinking, are there any major chat services that would go down if a particular AWS/GCP/etc data centre went down? You don't want your service to go down, plus your team's comms at the same time.

I'm impressed but Amazon Chime still appears to be working right now. It's sad because this is the one service that could go down and be a net benefit.

Re: AWS us-east-1 outage

#355

Post by jacquesm »

tomrod wrote: post #276
melony wrote: post #243

Earlier quoted context omitted.

And the guy who doesn't take responsibility gets promoted. Employees are not responsible for failures of management to set a good culture.

Not in healthy organizations, they don't.

Once you reach a certain size there are surprisingly few healthy organization, most of them turn into externalization engines with 4 beats per year.

Re: AWS us-east-1 outage

#356

Post by electroly »

jedberg wrote: post #241
dekhn wrote: post #146

Earlier quoted context omitted.

if you cannot access the control plane to create or destroy resources, it is down (partial availability). The jobs that are running are basically zombies.

Seems like the API is still working and so is auto scaling. So they aren’t really zombies. Partial availability isn’t the same as no availability.

The API is NOT working -- it may not have been listed on the service health dashboard when you posted that, but it is now. We haven't been able to launch an instance at all, and we are continuously trying. We can't even start existing instances.

Re: AWS us-east-1 outage

#357

Post by Enginerrrd »

Spivak wrote: post #237
JohnJamesRambo wrote: post #112

Earlier quoted context omitted.

> Goodhart's Law is expressed simply as: “When a measure becomes a target, it ceases to be a good measure.” It’s very frustrating. Why even have them?

Because "uptime" and "nines" became a marketing term. Simple as that. But the problem is that any public-facing measure of availability becomes a defacto marketing term.

The older I get the more I hate marketers. The whole field stands on the back of war-time propaganda research and it sure feels like it's the cause of so much rot in society.

Re: AWS us-east-1 outage

#358

Post by john37386 »

zegl wrote: post #13

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

It starts to show issues now. I agree that it was a bit long before we can get real visibility on the incident.

Re: AWS us-east-1 outage

#359

Post by jhugo »

blahyawnblah wrote: post #163
temuze wrote: post #113

Friends tell friends to pick us-east-2. Virginia is for lovers, Ohio is for availability.

Lots of services are only in us-east-1. The sso system isn't working 100% right now so that's where I assume it's hosted.

You can pick the region for SSO — or even use multiple. Ours is in ap-southeast-1 and working fine — but then the console that it signs us into is only partially working presumably due to dependencies on us-east-1.

Re: AWS us-east-1 outage

#360

Post by hinkley »

1-6 wrote: post #247
A4ET8a8uTh0 wrote: post #220

Earlier quoted context omitted.

Then we would drown in a sea of meetings and 'lessons learned' emails. There is a reason for post-mortems, but there has to be balance.

I find post-mortems interesting to read through especially when it’s not my fault. Most of them would probably be routine to read through but there are occasional ones that make me cringe or laugh. Post-mortems can sometime be thought of like safety training. There is a big imbalance of time dedicated to learning proper safety handling just for those small incidences.

Does Disney still play the "Instructional Videos" series starring Goofy where he's supposed to be teaching you how to do something and instead we learn how NOT to do something? Or did I just date myself badly?
Post reply on HN