Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

441–450 of 1001 posts

Re: AWS us-east-1 outage

#441

Post by mountainofdeath »

mrweasel wrote: post #211
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

That’s idiotic, the service is down regardless. If you foster that kind of culture, why have a status page at all? It make AWS engineers look stupid, because it looks like they are not monitoring their services.

The status page is as much a political tool as a technical one. Giving your service a non-green state makes your entire management chain responsible. You don't want to be one that upsets some VPs advancement plans.

Re: AWS us-east-1 outage

#442

Post by PragmaticPulp »

mbordenet wrote: post #416
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

When I worked for AMZN (2012-2015, Prime Video & Outbound Fulfillment), attempting to sweep issues under the rug was a clear path to termination. The Correction-Of-Error (COE) process can work wonders in a healthy, data-driven, growth-mindset culture. I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord? Blame deflection is a recipe for repeat outages and unhappy customers.

> I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord?

Entirely possible, and something I've always suspected.

Re: AWS us-east-1 outage

#443

Post by dekhn »

jedberg wrote: post #176
dekhn wrote: post #122

Earlier quoted context omitted.

Sure, but... that just raises more questions :) Taken literally what you are saying is the service could be down and an executive could override that, preventing them for paying customers for a service outage, even if the service did have an outage and the customer could prove it (screenshots, metrics from other cloud providers, many different folks see it). I'm sure there is some subtlety to this, but it does mean t…

Like I said I never worked there and this is all hearsay but there is a lot of nuance here being missed like partial outages.

This is no longer a partial outage. The status page reports elevated API error rates, DynamoDB issues, EC2 API error rates, and my company's monitoring is significantly affected (IE, our IT folks can't tell us what isn't working) and my AWS training class isn't working either.

If this needed a CEO to eventually get around to pressing a button that said "show users the actual information about a problem" that reflects poorly on amazon.

Re: AWS us-east-1 outage

#444

Post by amzn-throw »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

It's popular to upvote this during outages, because it fits a narrative.

The truth (as always) is more complex:

* No, this isn't the broad culture. It's not even a blip. These are EXCEPTIONAL circumstances by extremely bad teams that - if and when found out - would be intervened dramatically.

* The broad culture is blameless post-mortems. Not whose fault is it. But what was the problem and how to fix it. And one of the internal "Ten commandments of AWS availability" is you own your dependencies. You don't blame others.

* Depending on the service one customer's experience is not the broad experience. Someone might be having a really bad day but 99.9% of the region is operating successfully, so there is no reason to update the overall status dashboard.

* Every AWS customer has a PERSONAL health dashboard in the console that should indicate their experience.

* Yes, VP approval is needed to make any updates on the status dashboard. But that's not as hard as it may seem. AWS executives are extremely operation-obsessed, and when there is an outage of any size are engaged with their service teams immediately.

Re: AWS us-east-1 outage

#445

Post by DataGata »

xtracto wrote: post #390
anovikov wrote: post #168

Haha my developer called me in panic telling that he crashed Amazon - was doing some load tests with Lambda

How can you own a developer? is it expensive to buy one?

Don't get nitty about saying "my X". People say "my plumber" or "my hairstylist" or whatever all the time.

Re: AWS us-east-1 outage

#446

Post by treis »

bandyaboot wrote: post #415
NikolaeVarius wrote: post #370

Earlier quoted context omitted.

> Should I be penalized if an upstream dependency, owned by another team, fails? Yes > Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? Yes

Where does this mindset end? Do I lack due diligence by choosing to accept that the cpu microcode on the system I’m deploying to works correctly?

You choose the CPU and you choose what happens in a failure scenario. Part of engineering is making choices that meet the availability requirements of your service. And part of that is handling failures from dependencies.

That doesn't extend to ridiculous lengths but as a rule you should engineer around any single point of failure.

Re: AWS us-east-1 outage

#447

Post by throwaway82931 »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

This fits with everything I've heard about terrible code quality at Amazon and engineers working ridiculous hours to close tickets any way they can. Amazon as a corporate entity seems to be remarkably distrustful of and hostile to its labor force.

Re: AWS us-east-1 outage

#448

Post by NikolaeVarius »

bandyaboot wrote: post #415
NikolaeVarius wrote: post #370

Earlier quoted context omitted.

> Should I be penalized if an upstream dependency, owned by another team, fails? Yes > Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? Yes

Where does this mindset end? Do I lack due diligence by choosing to accept that the cpu microcode on the system I’m deploying to works correctly?

Yes? If you are worried about CPU microcode failing, then you do a NASA and have multiple CPU arch's doing calculations in a voting block. These are not unsolved problems.

Re: AWS us-east-1 outage

#449

Post by bee_rider »

tssva wrote: post #362
shepherdjerred wrote: post #143

Earlier quoted context omitted.

Does this imply Virginia is Godless?

Virginia's actual motto is "Sic semper tyrannis". What's more tyrannical than an omnipotent being that will condemn you to eternal torment if you don't worship them and follow their laws.

Virginia and Massachusetts have surprisingly aggressive mottoes (MA is: "By the sword we seek peace, but peace only under liberty", which is really just a fancy way of saying "don't make me stab a tyrant," if you think about it). It probably makes sense, though, given that they came up with them during the revolutionary war.

Re: AWS us-east-1 outage

#450

Post by blahyawnblah »

JamesAdir wrote: post #405

noob question: Aren't companies using several regions for availability and redundancy?

Ideally, yes. In practice, most are hosted in a single region but with multiple availability zones (this is called high availability). What you're talking about is fault tolerance (across multiple regions). That's harder to implement and costs more.
Post reply on HN