Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

411–420 of 1001 posts

Re: AWS us-east-1 outage

#411

Post by crmd »

JackFr wrote: post #377
SQueeeeeL wrote: post #293

Earlier quoted context omitted.

Would they? Having 3 outages in a year sounds like an organization problem. Not enough safeguards to prevent very routine human errors. But instead of worrying about that we just assign a guy to take the fall

Well if John caused 3 outages and and his peers Sally and Mike each caused 0, it's worth taking a deeper look. There's a real possibility he's getting screwed by a messed up org, also he could be doing slapdash work or he seriously might not undertsand the seriousness of an outage.

John’s team might also be taking more calculated risks and running circles around Sally and Mike’s teams with respect to innovation and execution. If your organization categorically punishes failures/outages, you end up with timid managers that are only playing defense, probably the opposite of what the leadership team wants.

Re: AWS us-east-1 outage

#413

Post by mym1990 »

jrootabega wrote: post #275
maximedupre wrote: post #227

Earlier quoted context omitted.

Or just take responsibility. People will respect you for doing that and you will demonstrate leadership.

Cynical/realist take: Take responsibility and then hope your bosses already love you, you can immediately both come with a way to prevent it from happening again, and convince them to give you the resources to implement it. Otherwise your responsibility is, unfortunately, just blood in the water for someone else to do all of that to protect the company against you and springboard their reputation on the descent of yo…

This seems like an absolutely horrid way of working or doing 'office politics'.

Re: AWS us-east-1 outage

#414

Post by abledon »

kumarakn wrote: post #336
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

I worked at Walmart Technology. I bravely wrote post mortem documents owning the fault of my team (100+ people), owning both technically and also culturally as their leader. I put together a plan to fix it and executed it. Thought that was the right thing to do. This happend two times in my 10 year career there. Both times I was called out as a failure in my performance eval. Second time, I resigned and told them to…

is it just 'ceremony' to be called out on those things? (even if it is actually a positive sum total)

Re: AWS us-east-1 outage

#415

Post by bandyaboot »

NikolaeVarius wrote: post #370
88913527 wrote: post #352

Earlier quoted context omitted.

Should I be penalized if an upstream dependency, owned by another team, fails? Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? These are real problems in the micro-services world, especially since I own UI and there are dozens of teams pumping out services, and I'm at the mercy of all of them. The best I can do is gracefully fail when services don't function in a healthy…

> Should I be penalized if an upstream dependency, owned by another team, fails? Yes > Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? Yes

Where does this mindset end? Do I lack due diligence by choosing to accept that the cpu microcode on the system I’m deploying to works correctly?

Re: AWS us-east-1 outage

#416

Post by mbordenet »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

When I worked for AMZN (2012-2015, Prime Video & Outbound Fulfillment), attempting to sweep issues under the rug was a clear path to termination. The Correction-Of-Error (COE) process can work wonders in a healthy, data-driven, growth-mindset culture. I wonder if the ex-Amazonian you're referring to did not leave AMZN by their own accord?

Blame deflection is a recipe for repeat outages and unhappy customers.

Re: AWS us-east-1 outage

#417

Post by stephenr »

itisit wrote: post #406
stephenr wrote: post #342

So, we're getting failures (for customers) trying to use amazon pay from our site. AFAIK there is no "status page" for Amazon Pay, but the rest of Amazon's services seem to be a giant Rube Goldberg machine so it's hard to imagine this isn't too.

http://status.mws.amazon.com/

Thanks.. seems to be about as accurate as the regular AWS status board is..

Re: AWS us-east-1 outage

#418

Post by iso1210 »

john37386 wrote: post #387

It seems a bit long to fix! They probably paint themselves in a corner just like facebook few weeks ago. This make me think; Could it be that one day the internet will have a total global outage and it will take few days to recover?

I'm not sure how you get a total global outage in a distributed system. Lets say a major transit provider (Century link for example) advertises traffic as a "go via me", but then drops the traffic, lets also assume it drops the costs of routes to pretty much zero. That would certainly have a major effect, until their customers/peers stop their peers.

That might be tricky if they are remote and not on the same AS as their router access points and have no completely out of band access, but you're still talking hours at most.

Re: AWS us-east-1 outage

#420

Post by bgribble »

hinkley wrote: post #325
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

I am finding that I have a very bimodal response to "He did it". When I write an RCA or just talk about near misses, I may give you enough details to figure out that Tom was the one who broke it, but I'm not going to say Tom on the record anywhere, with one extremely obvious exception. If I think Tom has a toxic combination of poor judgement, Dunning-Kruger syndrome, and a hint of narcissism (I'm not sure but I may b…

To me, the point of "blameless" PM is not to hide the identity of the person who was closest to the failure point. You can't understand what happened unless you know who did what, when.

"Blameless" to me means you acknowledge that the ultimate problem isn't that someone made a mistake that caused an outage. The problem is that you had a system in place where someone could make a single mistake and cause an outage.

If someone fat-fingers a SQL query and drops your database, the problem isn't that they need typing lessons! If you put a DBA in a position where they have to be typing SQL directly at a production DB to do their job, THAT is the cause of the outage, the actual DBA's error is almost irrelevant because it would have happened eventually to someone.

Post reply on HN