Netflix Chaos Monkey Upgraded

#1

Post by dustinmoris » Wed, Oct 19, 2016, 1:48 PM UTC

Netflix Chaos Monkey Upgraded
techblog.netflix.com

Re: Netflix Chaos Monkey Upgraded

#2

Post by moondev » Wed, Oct 19, 2016, 2:34 PM UTC

This is awesome. Since Chaos Monkey now leverages spinnaker you can run it against clouddriver provider. Looking forward to trying this out with kubernetes. I believe spinnaker treats namespaces as regions. Eventually it would be cool to simulate masters or other entire federated clusters going down to test kubernetes scheduling resilience

Re: Netflix Chaos Monkey Upgraded

#3

Post by ticklemyelmo » Wed, Oct 19, 2016, 2:37 PM UTC

It looks like it's working too well. The site is unreachable for me.

Re: Netflix Chaos Monkey Upgraded

#4

Post by netsec_burn » Wed, Oct 19, 2016, 2:39 PM UTC

ticklemyelmo wrote: post #3

It looks like it's working too well. The site is unreachable for me.

It's up.

Re: Netflix Chaos Monkey Upgraded

#5

Post by e1g » Wed, Oct 19, 2016, 2:40 PM UTC

Another useful tool is https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).

Re: Netflix Chaos Monkey Upgraded

#6

Post by yalooze » Wed, Oct 19, 2016, 2:44 PM UTC

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

Re: Netflix Chaos Monkey Upgraded

#7

Post by dest » Wed, Oct 19, 2016, 2:48 PM UTC

ticklemyelmo wrote: post #3

It looks like it's working too well. The site is unreachable for me.

down : Secure Connection Failed

The connection to techblog.netflix.com was interrupted while the page was loading.

Re: Netflix Chaos Monkey Upgraded

#8

Post by jsingleton » Wed, Oct 19, 2016, 2:52 PM UTC

Has anyone else deployed a Chaos Monkey in production?

I can imagine it would be a tough sell to the CEO. :)

Re: Netflix Chaos Monkey Upgraded

#9

Post by fred256 » Wed, Oct 19, 2016, 2:53 PM UTC

dest wrote: post #7

ticklemyelmo wrote: post #3

It looks like it's working too well. The site is unreachable for me.

down : Secure Connection Failed The connection to techblog.netflix.com was interrupted while the page was loading.

It's only available on http, not https.

Re: Netflix Chaos Monkey Upgraded

#10

Post by jsingleton » Wed, Oct 19, 2016, 2:55 PM UTC

yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.