Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix Chaos Monkey Upgraded

techblog.netflix.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: Netflix Chaos Monkey Upgraded

#41

Post by fyshotshot »

saulrh wrote: post #28
yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.

Seems to me those kind of issues would be a good way to test your monitoring. For instance verifying that the appropriate people are notified or automatic action is taken when within a reasonable amount of time after the CPU on the box starts spiking.

Re: Netflix Chaos Monkey Upgraded

#42

Post by birdman3131 »

klapinat0r wrote: post #14
jsingleton wrote: post #8

Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)

How so? The benefits are worth it, and I doubt any CEO will be argue against having fault tolerant code :) You catch bugs, and no one says you can't run Chaos Monkey in staging or a similar environment if it really is a tough sell.

The drawbacks of potentially causing downtime and therefore having the potential to drive away customers as well as obtain an image of unreliability can be much more damaging than not using it in the first place. Customer image means quite a bit.

Re: Netflix Chaos Monkey Upgraded

#43

Post by benhoyt »

jsingleton wrote: post #10
yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)

Re: Netflix Chaos Monkey Upgraded

#44

Post by timf »

e1g wrote: post #5

Another useful tool is https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).

Blockade is another tool to check out in this area: http://blockade.readthedocs.io/en/latest/

Re: Netflix Chaos Monkey Upgraded

#45

Post by iagooar »

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that".

While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

Re: Netflix Chaos Monkey Upgraded

#46

Post by devonkim »

benhoyt wrote: post #43
jsingleton wrote: post #10

Earlier quoted context omitted.

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)

"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.

Re: Netflix Chaos Monkey Upgraded

#47

Post by corobo »

scrollaway wrote: post #22
Thaxll wrote: post #13

Earlier quoted context omitted.

Every time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.

"Every time"? If one random guy's complaining about a URL being unreachable and you're already seeing a pattern... is it at all possible the users aren't at fault?

To be fair it does happen frequently enough. I wouldn't say every time but often on Netflix posts. A bit of a sampling for you:

https://news.ycombinator.com/item?id=12269411

https://news.ycombinator.com/item?id=12217900

https://news.ycombinator.com/item?id=12038367

https://news.ycombinator.com/item?id=11771714

Re: Netflix Chaos Monkey Upgraded

#48

Post by Thaxll »

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

Netflix had a worldwide outage few weeks ago that last hours so yes they need those kind of tools :)

Re: Netflix Chaos Monkey Upgraded

#49

Post by RBerenguel »

benhoyt wrote: post #43
jsingleton wrote: post #10

Earlier quoted context omitted.

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)

If deployment is automated and "clean", images get baked into machines and they just start. For instance, we use Ansible against the machine itself on boot, so we don't really need ssh access to it: everything is automated (but we keep it open to troubleshoot anything that may happen)

Re: Netflix Chaos Monkey Upgraded

#50

Post by agentgt »

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

The joke is funny but it is actually shockingly difficult to make nodes kill themselves instead of doing something for more malignant which is the typical case for bugs. That is a zombie just like in real life can be worse than a dead corpse (since we are being funny and all). Chaos Monkey shoots two in the head.

I have had issues killing errant JVMs and Rackspace nodes (yes sadly we are still on Rackspace).

I can understand why 2.0 is much more focused given the plethora of monitoring solutions.

Post reply on HN