Netflix Chaos Monkey Upgraded

Post by drdrey » Wed, Oct 19, 2016, 6:18 PM UTC

saulrh wrote: post #28

yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.

Post by brianwawok » Wed, Oct 19, 2016, 6:47 PM UTC

rlau26 wrote: post #72

"Chaos Monkey even periodically terminates itself." Whoa, that's meta.

Post by tormeh » Wed, Oct 19, 2016, 6:58 PM UTC

aaronblohowiak wrote: post #64

Beltiras wrote: post #21

Every time this is in the news I get a feeling of awe for operations teams having confidence enough to deploy this. I've usually been in small teams with the feature mill factor turned up way too high.

If you don't use a tool like this, entropy will take care of taking your machines down for you. Only then, it won't be a regularly rehearsed part of "normal operations" so you might find yourself up creek without a paddle.

Post by jpolit » Wed, Oct 19, 2016, 7:58 PM UTC

flgr wrote: post #31

JoachimSchipper wrote: post #23

Earlier quoted context omitted.

Yes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/ , because Dan Luu's site is always excellent.) "We rewrote the service for improved maintainability" seems an important part of this blog post.

I wonder if they have a service that automatically detects all those failures and terminates the entire node instead?

Post by cpeterso » Wed, Oct 19, 2016, 8:23 PM UTC

dsl wrote: post #76

cpeterso wrote: post #60

Beyond external termination services like Chaos Monkey, what are good examples of software that purposely increase internal nondeterminism or failure injection in production? Go has race detector mode, but it is an optional debug feature with a performance cost. The Linux kernel's jiffy clock starts counting from -5 minutes so drivers must handle clock rollover correctly because it's not a uncommon "once every 48 day…

Go intentionally introduces randomness when reading maps so developers don't write code dependent on order.