Netflix Chaos Monkey Upgraded • Hacker News Board

Re: Netflix Chaos Monkey Upgraded

#61

Post by aaronblohowiak » Wed, Oct 19, 2016, 5:03 PM UTC

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

We have different tools for different kinds of failures.

Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we are resilient to region failure.

Most of the developer-induced pain (which is most frequent source of pain) happens at the service level -- a bad code push that somehow made it through canary, accidentally doing something you shouldn't, misconfiguring something, etc. For tolerating service-level failures, we use different tools that minimize the fallout of the failure injection. Specifically, FIT (and the soon to be revealed ChAP.) These tools allow us to be more surgical in our injection of failure and tie that into our telemetry solutions.

We only inject failures we expect to be resilient to. Sadly, that is a subset of the failures that people cause ;)

Re: Netflix Chaos Monkey Upgraded

#62

Post by stgnet » Wed, Oct 19, 2016, 5:04 PM UTC

Inspired by chaos monkey, I introduced malloc chaos mechanism into our codebase: https://reviewboard.asterisk.org/r/4463/

Although designed originally to catch places where malloc failure wasn't being handled, it can also be used to randomly trigger other off-nominal portions of the code that might not otherwise be tested.

Re: Netflix Chaos Monkey Upgraded

#63

Post by dberg » Wed, Oct 19, 2016, 5:06 PM UTC

devonkim wrote: post #46

benhoyt wrote: post #43

Earlier quoted context omitted.

What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)

"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.

So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ?

Edit: Sorry responded to wrong parent, sigh.

Re: Netflix Chaos Monkey Upgraded

#64

Post by aaronblohowiak » Wed, Oct 19, 2016, 5:15 PM UTC

Beltiras wrote: post #21

Every time this is in the news I get a feeling of awe for operations teams having confidence enough to deploy this. I've usually been in small teams with the feature mill factor turned up way too high.

If you don't use a tool like this, entropy will take care of taking your machines down for you. Only then, it won't be a regularly rehearsed part of "normal operations" so you might find yourself up creek without a paddle.

Re: Netflix Chaos Monkey Upgraded

#65

Post by sametmax » Wed, Oct 19, 2016, 5:26 PM UTC

dberg wrote: post #63

devonkim wrote: post #46

Earlier quoted context omitted.

"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.

So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ? Edit: Sorry responded to wrong parent, sigh.

It's just a matter of scale. If you are at the scale of netflix, vm are probably too complex black boxes, and logs output somewhere else anyway. Plus the problem may involve the interraction of several vm, or the network, or other composants together.

Re: Netflix Chaos Monkey Upgraded

#66

Post by FireBeyond » Wed, Oct 19, 2016, 5:41 PM UTC

jsingleton wrote: post #10

yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

I actually agree with you about immutable infrastructure, as I work at implementing it.

But that's dangerously close to the "One True Way". Which is certainly not the case - so much of this is evolving, and a wide variety of situations and circumstances.

Re: Netflix Chaos Monkey Upgraded

#67

Post by kevan » Wed, Oct 19, 2016, 5:45 PM UTC

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

Developers cause software downtime, but they usually don't cause infrastructure downtime, which is what Chaos Monkey does. Cloud VMs have failure rates somewhere around 1-2% depending on who you ask. This is low enough that you can ignore it most of the time, but it'll come back and bite you hard later. Chaos Monkey artificially forces that failure rate high enough that you'll notice problems immediately and fix them before they become too engrained into your architecture.

Re: Netflix Chaos Monkey Upgraded

#68

Post by btown » Wed, Oct 19, 2016, 5:49 PM UTC

aaronblohowiak wrote: post #25

andrewguenther wrote: post #11

Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?

Resource exhaustion manifests as latency or failure. We inject latency and failure using FIT, so we can limit the "blast radius". When you are testing these failure modes, you are more testing the interaction between micro services, and this requires a bit more precision and sophistication. Source: I'm on the Chaos team here at Netflix.

For reference, FIT: http://techblog.netflix.com/2014/10/fit-failure-injection-te...

Re: Netflix Chaos Monkey Upgraded

#69

Post by kpwagner » Wed, Oct 19, 2016, 6:00 PM UTC

Holy fuck that is some small font size. And the paragraphs aren't in paragraph tags... just hanging out between 's with some
's to keep them company.

Re: Netflix Chaos Monkey Upgraded

#70

Post by jackweirdy » Wed, Oct 19, 2016, 6:06 PM UTC

gkop wrote: post #52

haneefmubarak wrote: post #35

Earlier quoted context omitted.

AFAIR Jepsen is only meant to test the CAP aspects of a system, not how it behaves otherwise.

Parent may have meant that it would be cool to see a write up on how this tool can be used to simulate network conditions between nodes in a Jepsen test using only containers on a single docker host.

Yep, that's what I meant.