Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

441–450 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#441

Post by evilDagmar »

cjbprime wrote: post #422
mulmen wrote: post #400

Earlier quoted context omitted.

Agreed, especially regarding the culture but isn't this pretty much the same explanation they gave a few years ago when something similar happened? I seem to recall an EC2 or S3 outage a few years ago that boiled down to an engineer pushing out a patch that broke an entire region when it was supposed to be a phased deployment. I could be mis-remembering that but it's important that these lessons be applied across the…

Pretty sure that one was a Microsoft Azure outage. (Source: am a self-identified post-mortems connoisseur. :)

Not a bad plan. If you don't make enough mistakes on your own, ya gotta learn from the mistakes of others as a preventative.

Re: Summary of the Amazon S3 Service Disruption

#442

Post by jsudhams »

halomru wrote: post #351
jegoodwin3 wrote: post #335

Earlier quoted context omitted.

I agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?

>We know the internet has a single point of failure. It has? I have yet to see the day where I can neither reach my email provider nor Google nor Hackernews. My local provider might screw up occasionally, or some number of of websites go unreachable for whatever reason. But I fail to come up with anything short of cutting multiple see cables that causes more than 50% of servers to be unreachable to more than 50% of u…

https://en.wikipedia.org/wiki/2008_submarine_cable_disruptio...

Re: Summary of the Amazon S3 Service Disruption

#445

Post by jsudhams »

gigatexal wrote: post #188
duskwuff wrote: post #185

Earlier quoted context omitted.

FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.

hmm, that's kinda cool. I'm in a MS shop and I don't know if SMSS has the same feature. My manager just looked at me and said "welp, go restore the table and be more careful next time." I was a new DBA at the time, still, kinda new.

USE BEGIN TRANS as mentioned above

Re: Summary of the Amazon S3 Service Disruption

#446

Post by jsudhams »

gigatexal wrote: post #188
duskwuff wrote: post #185

Earlier quoted context omitted.

FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.

hmm, that's kinda cool. I'm in a MS shop and I don't know if SMSS has the same feature. My manager just looked at me and said "welp, go restore the table and be more careful next time." I was a new DBA at the time, still, kinda new.

USE BEGIN TRANS as mentioned above

Re: Summary of the Amazon S3 Service Disruption

#447

Post by grogers »

snewman wrote: post #131
conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enoug…

This is precisely why cells (alluded to in the write-up) are beneficial. If the size of a cell is bounded and you scale by adding more cells, testing the breaking point of the largest cell becomes an easier problem. There is still usually a layer that spans across all cell boundaries, which is what then becomes hard to test at prod scale (so you make that as simple as possible)

Re: Summary of the Amazon S3 Service Disruption

#448

Post by mrep »

pinko wrote: post #67
savanaly wrote: post #52

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

There are people who can still do fleet wide root access commands and I don't think that type of thing will ever be removed (just very restricted) for this exact type of situation.

Re: Summary of the Amazon S3 Service Disruption

#449

Post by noisy_boy »

magd wrote: post #42

Oh, that interview question. “Tell me about something you broke in your last job"

Due to youth, totally misplaced confidence and poor access rights regime, ran an untested script in production causing it to fork uncontrollably requiring reboot.

Re: Summary of the Amazon S3 Service Disruption

#450

Post by mml »

patio11 wrote: post #432
justin_oaks wrote: post #418

Earlier quoted context omitted.

It is unreasonable for me to think that company owners should have the spine to say, "We take the decision to fire someone very seriously. We'll take your comments under consideration, but we retain sole discretion over such decisions." It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.
Post reply on HN