Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

491–500 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#491

Post by pron »

jegoodwin3 wrote: post #335
djsumdog wrote: post #222

Earlier quoted context omitted.

Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this." Tests and configuration scripts don't preven…

I agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?

Amazon do formally verify AWS (they use TLA+), which is probably why this failure is a human error. Of course, you could expand the formal analysis of the system to include all possible operator interactions, but you'll need to draw the line at some point. NASA certainly makes human errors that result in catastrophic failures. The Challenger disaster was also a result of human error to a large degree[1]; to quote Wikipedia: "The Rogers Commission found NASA's organizational culture and decision-making processes had been key contributing factors to the accident, with the agency violating its own safety rules."

[1]: https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disas...

Re: Summary of the Amazon S3 Service Disruption

#492

Post by pron »

icebraining wrote: post #382
gmac wrote: post #341

Earlier quoted context omitted.

NASA doesn't make these sorts of mistakes https://www.wired.com/2010/11/1110mars-climate-observer-repo...

Almost twenty years ago, though.

Well, they've had plenty of opportunities to learn from their mistakes; Amazon hasn't had this long.

Re: Summary of the Amazon S3 Service Disruption

#493

Post by gadders »

ascorbic wrote: post #482
patio11 wrote: post #432

Earlier quoted context omitted.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

The BBC staff have a term for the way the corporation almost does this: "deputy heads will roll"

I always thought that was more a cynical take on the fact that the top guy was protected, rather than underlings.

Re: Summary of the Amazon S3 Service Disruption

#494

Post by marklyon »

mml wrote: post #450
patio11 wrote: post #432

Earlier quoted context omitted.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.

I know one person who does this for real estate developers. He gets involved in contentious projects early on, goes to community meetings, offers testimony before the city council, etc. When construction gets going and people inevitably get pissed about some aspect of the project, he gets publicly fired to deflect the blame while the project moves on. Have seen it happen on three different projects in two cities now and, somehow, nobody catches on.

Re: Summary of the Amazon S3 Service Disruption

#496

Post by jacquesm »

aanm1988 wrote: post #487
jacquesm wrote: post #398

Earlier quoted context omitted.

What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes. Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit). This usually then caused the custome…

Did you ever insist that Frans must be fired and refuse to accept the "we didn't mean it?" Cause that sounds pretty great.

Well, by then he was fired... :)

Re: Summary of the Amazon S3 Service Disruption

#497

Post by jacquesm »

pieterr wrote: post #484
jacquesm wrote: post #398

Earlier quoted context omitted.

What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes. Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit). This usually then caused the custome…

Koot en Bie - Mannen Bellen https://open.spotify.com/track/0tzlDFcFSNRUaaPnINWo8B

That's one deserved upvote :)

Re: Summary of the Amazon S3 Service Disruption

#499

Post by Exuma »

brongondwana wrote: post #483
Exuma wrote: post #190

Imagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP

https://blog.fastmail.com/2011/05/15/outage-report-a-cascade... I was the guy who deployed the update to every one of our servers as I walked out the door for the day. So I know what this feels like. You learn to get through it fast, because there's no other choice, you're almost always the best placed person to clean up the mess. And one day you can look back and the scars have healed.

Damn! Thanks for sharing your story, it is a great one.

Re: Summary of the Amazon S3 Service Disruption

#500

Post by ufmace »

problems wrote: post #263
Bartweiss wrote: post #240

Earlier quoted context omitted.

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

This is true in some cases, but not when mitigations aren't practiced properly - it's not the fat fingered user who should be fired or retrained, but the designer or maintainer of the system that allowed it to become a serious issue. Look at the recent GitLab incident - one guy messed up and nuked a server. Okay, that happens sometimes, go to backups. Uh oh, all the backups are broken. Minor momentary problem just tu…

Giving people training in response to things like this always seemed a little strange to me - that particular person just got the most effective training the world has ever seen. If you look at it that way, you could say that everything Amazon spent responding to this was actually a training expense for this particular person and team. After you've already done that, it seems silly to make them sit through some online quiz or PowerPoint by a supposed guru and think you're accomplishing anything.
Post reply on HN