Every time this is in the news I get a feeling of awe for operations teams having confidence enough to deploy this. I've usually been in small teams with the feature mill factor turned up way too high.
Netflix Chaos Monkey Upgraded
21–30 of 89 posts
Re: Netflix Chaos Monkey Upgraded
#22It looks like it's working too well. The site is unreachable for me.
Every time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.
"Every time"?
If one random guy's complaining about a URL being unreachable and you're already seeing a pattern... is it at all possible the users aren't at fault?
Re: Netflix Chaos Monkey Upgraded
#23Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
Yes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/, because Dan Luu's site is always excellent.)
"We rewrote the service for improved maintainability" seems an important part of this blog post.
Re: Netflix Chaos Monkey Upgraded
#24Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
Maybe internally such situations translate to a termination anyway.
Re: Netflix Chaos Monkey Upgraded
#25Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
Resource exhaustion manifests as latency or failure. We inject latency and failure using FIT, so we can limit the "blast radius". When you are testing these failure modes, you are more testing the interaction between micro services, and this requires a bit more precision and sophistication.
Source: I'm on the Chaos team here at Netflix.
Re: Netflix Chaos Monkey Upgraded
#26Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
perhaps it had too much liability.
Re: Netflix Chaos Monkey Upgraded
#27Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)
Yes.
The "sell" can be tricky for some people, until your first production issue. Never let a crisis go to waste.
Machines go away all the time in the cloud. This tool increases the frequency so you can ensure your system handles it gracefully.
Some people believe their system can tolerate this class of failures, but without continuous validation, that is more of a hope than a certainty.
Re: Netflix Chaos Monkey Upgraded
#28I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...
If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.
Re: Netflix Chaos Monkey Upgraded
#29[deleted]
Re: Netflix Chaos Monkey Upgraded
#30Every time this is in the news I get a feeling of awe for operations teams having confidence enough to deploy this. I've usually been in small teams with the feature mill factor turned up way too high.
I plan to run on it on staging while running load test suites. Not brave enough to run it in production yet