Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

471–480 of 1001 posts

Re: AWS us-east-1 outage

#471

Post by hinkley »

bgribble wrote: post #420
hinkley wrote: post #325

Earlier quoted context omitted.

I am finding that I have a very bimodal response to "He did it". When I write an RCA or just talk about near misses, I may give you enough details to figure out that Tom was the one who broke it, but I'm not going to say Tom on the record anywhere, with one extremely obvious exception. If I think Tom has a toxic combination of poor judgement, Dunning-Kruger syndrome, and a hint of narcissism (I'm not sure but I may b…

To me, the point of "blameless" PM is not to hide the identity of the person who was closest to the failure point. You can't understand what happened unless you know who did what, when. "Blameless" to me means you acknowledge that the ultimate problem isn't that someone made a mistake that caused an outage. The problem is that you had a system in place where someone could make a single mistake and cause an outage. If…

Naming someone is how you discover that not everyone in the organization believes in Blamelessness. Once it's out it's out, you can't put it back in.

It's really easy for another developer to figure out who I'm talking about. Managers can't be arsed to figure it out, or at least pretend like they don't know.

Re: AWS us-east-1 outage

#472

Post by alpha_squared »

mystcb wrote: post #213

Not sure it helps, but got this update from someone inside AWS a few moments ago. "We have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also services like S3. We are actively working towards recovery."

That's a copy-paste, we got the same thing from our AWS contact. It's just enough info to confirm there's an issue, but not enough to give any indication on the scope or timeline to resolution.

Re: AWS us-east-1 outage

#475

Post by longhairedhippy »

woshea901 wrote: post #466

N. Virginia consistently has more problems than other zones. Is it possible this zone is also hosting government computers/could it be a more frequent target for this reason?

The real reason is us-east-1 was the first and by far the biggest region, the same reason that new services always launch there but other regions are are not necessarily required (some services have to launch in every region).

The us-east-1 region is consistently pushing the limits of scale for the AWS services, thus is has way more problems than other regions.

Re: AWS us-east-1 outage

#476

Post by sneak »

zegl wrote: post #13

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

It's widespread industry knowledge now that AWS is publicly dishonest about downtime.

When the biggest cloud provider in the world is famous for gaslighting, it sets expectations for our whole industry.

It's fucking disgraceful that they tolerate such a lack of integrity in their organization.

Re: AWS us-east-1 outage

#477

Post by beamatronic »

DarthNebo wrote: post #473
endisneigh wrote: post #31

Azure, Google Cloud, AWS and others need to have a “Status alliance” where they determine the status of each of their services by a quorum using all cloud providers. Status pages are virtually useless these days

https://en.wikipedia.org/wiki/Mexican_standoff

It's not the prisoner's dilemma?

Re: AWS us-east-1 outage

#478

Post by javajosh »

88913527 wrote: post #352
javajosh wrote: post #302

Earlier quoted context omitted.

I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…

Should I be penalized if an upstream dependency, owned by another team, fails? Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? These are real problems in the micro-services world, especially since I own UI and there are dozens of teams pumping out services, and I'm at the mercy of all of them. The best I can do is gracefully fail when services don't function in a healthy…

I wish you wouldn't talk about "penalization" as if it was something that comes from a source of authority. Your customers are depending on you, and you've let them down, and the reason that's bad has nothing to do with what your boss will do to you in a review.

The injustice that can and does happen is that you're explicitly given a narrow responsibility during development, and then a much broader responsibility during operation. This is patently unfair, and very common. For something like a failed uService you want to blame "the architect" that didn't anticipate these system level failures. What is the solution? Have plan b (and plan c) ready to go. If these services don't exist, then you must build them. It also implies a level of indirection that most systems aren't comfortable with, because we want to consume services directly (and for good reason) but reliability requires that you never, ever consume a service directly, but instead from an in-process location that is failure aware.

This is why reliable software is hard, and engineers are expensive.

Oh, and it's also why you generally do NOT want to defer the last build step to runtime in the browser. If you start combining services on both the client and server, you're in for a world of hurt.

Re: AWS us-east-1 outage

#480

Post by beamatronic »

smt88 wrote: post #62
endisneigh wrote: post #31

Azure, Google Cloud, AWS and others need to have a “Status alliance” where they determine the status of each of their services by a quorum using all cloud providers. Status pages are virtually useless these days

They can do this without an alliance. They very intentionally choose not to do it. Every major company has moved away from having accurate status pages.

In a broader world sense, we live in the post-truth era.
Post reply on HN