Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
Netflix Chaos Monkey Upgraded
11–20 of 89 posts
Re: Netflix Chaos Monkey Upgraded
#12It looks like it's working too well. The site is unreachable for me.
down : Secure Connection Failed The connection to techblog.netflix.com was interrupted while the page was loading.
Looks like some sequence of events convinces Firefox that HSTS is set, and it always rewrites the request as https (which is not supported by the server) after that.
Re: Netflix Chaos Monkey Upgraded
#13It looks like it's working too well. The site is unreachable for me.
Every time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.
Re: Netflix Chaos Monkey Upgraded
#14Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)
How so? The benefits are worth it, and I doubt any CEO will be argue against having fault tolerant code :)
You catch bugs, and no one says you can't run Chaos Monkey in staging or a similar environment if it really is a tough sell.
Re: Netflix Chaos Monkey Upgraded
#15Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)
Chaos Monkey is sort of like Advanced Continuous Deployment. Most shops are still struggling with the basics. You cant even think of trying to sell this running to the C-level until you've proven that you can at least walk (automated deployment and rollback).
I remember reading years and years ago about bandit algorithms... this kind of ops work is at a level that's found only in a few different companies.
Re: Netflix Chaos Monkey Upgraded
#16Another useful tool is https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).
I'd love to see this used in the Jepsen tests
Re: Netflix Chaos Monkey Upgraded
#17I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...
I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.
Everything goes through Spinnaker now, which in turn supports all clouddriver provides including aws, gcp, azure and kubernetes.
Resource limits should be set by instance type. It's more of an application level thing than infrastructure which is what chaos monkey is supposed to simulate
Re: Netflix Chaos Monkey Upgraded
#18I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...
Was wondering the same thing... I know in our environment, unexpected events on a box cause more problems than entire server failures. We design around servers coming in and out; specific processes failing in random ways is harder to design around.
Re: Netflix Chaos Monkey Upgraded
#19First, a shameless plug for an alternative implementation: https://github.com/BBC/chaos-lambda
Seems unfortunate that it requires the coupling with spinnaker - although i can see how it helps with the cluster definition features.
Edit: I'll add that we've been using the original chaos monkey and chaos lambda extensively in production for some time with very few problems.
Re: Netflix Chaos Monkey Upgraded
#20Another useful tool is https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).
What do you use for high availability with Docker?