Netflix Chaos Monkey Upgraded • Hacker News Board

Re: Netflix Chaos Monkey Upgraded

#81

Post by relix42 » Wed, Oct 19, 2016, 9:36 PM UTC

jsingleton wrote: post #8

Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)

In my opinion, it doesn't count unless its in production.

Why? Your customers use your production environment, not your test environment. Something will cause loss of an instance for you: * Mistaken termination * AWS retirement (and you missed the email) * Cable trip in the data center * *

So, vaccinate against the loss of an instance cratering your service. Give your prod environment a booster shot (with Chaos Monkey or something like it) every hour of every day. Then, when anything from the above list happens you're infrastructure handles it gracefully and without intervention. Continued booster shots ensure that this stability continues through config changes, software version changes, OS changes, tooling changes, etc.

I think the better question is "Why wouldn't you do this?"

Re: Netflix Chaos Monkey Upgraded

#82

Post by e1g » Thu, Oct 20, 2016, 1:43 AM UTC

brunoqc wrote: post #20

e1g wrote: post #5

Another useful tool is https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).

What do you use for high availability with Docker?

HA containers for us means smart orchestration tools. We did not want to lock ourselves into Docker-only infrastructure (even now rkt is a very compelling alternative), and wanted an orchestrator/scheduler that is focused entirely on that job. Outside of Swarm, Mesos & co appeared too intrusive, and Nomad is quite narrow in what it does. So we picked Kubernetes and are very happy with it.

Re: Netflix Chaos Monkey Upgraded

#83

Post by saryant » Thu, Oct 20, 2016, 3:52 AM UTC

dberg wrote: post #63

devonkim wrote: post #46

Earlier quoted context omitted.

"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.

So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ? Edit: Sorry responded to wrong parent, sigh.

At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical.

Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.

Re: Netflix Chaos Monkey Upgraded

#84

Post by caf » Thu, Oct 20, 2016, 3:56 AM UTC

I can see that Chaos Monkey adds selective pressure to ensure that systems evolve into a state where they can handle unexpected server outages.

But isn't there a danger that it also encourages maladaptions that come to rely on being regularly restarted by the Chaos Monkey? I'm particularly thinking that you might evolve a lot of resource leaks that go unnoticed so long as Chaos Monkey is on the job.

Re: Netflix Chaos Monkey Upgraded

#85

Post by caf » Thu, Oct 20, 2016, 4:00 AM UTC

cpeterso wrote: post #60

Beyond external termination services like Chaos Monkey, what are good examples of software that purposely increase internal nondeterminism or failure injection in production? Go has race detector mode, but it is an optional debug feature with a performance cost. The Linux kernel's jiffy clock starts counting from -5 minutes so drivers must handle clock rollover correctly because it's not a uncommon "once every 48 day…

The Linux kernel also recently added a debugging option that when enabled, instead of just probing a device it performs a probe / remove / probe sequence to ensure that device removal works.

Re: Netflix Chaos Monkey Upgraded

#86

Post by bbarn » Thu, Oct 20, 2016, 4:15 AM UTC

yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

I think it could be because it doesn't provide the value to the development teams. Why chase a memory leak or a CPU bug if it's just being caused by your fault testing app?

To prevent the negative effect of random machines dissapearing though.. that's a challenge that involves good ops, devs, even UI/UX I would imagine, and closer to something that users experience negatives because of in real life.

Re: Netflix Chaos Monkey Upgraded

#87

Post by vacri » Thu, Oct 20, 2016, 5:12 AM UTC

saryant wrote: post #83

dberg wrote: post #63

Earlier quoted context omitted.

So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ? Edit: Sorry responded to wrong parent, sigh.

At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical. Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.

Parsing logs isn't the only way to troubleshoot.

Re: Netflix Chaos Monkey Upgraded

#88

Post by flukus » Thu, Oct 20, 2016, 5:23 AM UTC

vacri wrote: post #87

saryant wrote: post #83

Earlier quoted context omitted.

At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical. Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.

Parsing logs isn't the only way to troubleshoot.

It's not even the best way, just sometimes it's the only way.

Re: Netflix Chaos Monkey Upgraded

#89

Post by Nitramp » Thu, Oct 20, 2016, 5:44 AM UTC

yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

To test a distributed system, there's not a lot of value to simulate all the many different conditions that can happen on a machine. From the system's perspective, you don't really care what happens on a single computer.

The conditions you need to simulate are (a) the machine being abruptly gone or (b) the machine still accepting requests, but being very slow in returning them and maybe (c) machine returning incorrect results. Seen from the outside, everything that can happen is usually (a) or (b), and with ECC memory and reasonable software (?) hopefully never (c).