Netflix Chaos Monkey Upgraded • Hacker News Board

Re: Netflix Chaos Monkey Upgraded

#51

Post by asadjb » Wed, Oct 19, 2016, 4:44 PM UTC

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

That rings so true. But on the flip side, it might be that because Netflix has a chaos monkey and all it's services need to be resilient to failures, accidental downtime isn't that big of an issue to them.

Their developers can still make the same mistakes that we do, but their architecture is better designed to handle that. Just a thought.

Re: Netflix Chaos Monkey Upgraded

#52

Post by gkop » Wed, Oct 19, 2016, 4:46 PM UTC

haneefmubarak wrote: post #35

jackweirdy wrote: post #16

Earlier quoted context omitted.

I'd love to see this used in the Jepsen tests

AFAIR Jepsen is only meant to test the CAP aspects of a system, not how it behaves otherwise.

Parent may have meant that it would be cool to see a write up on how this tool can be used to simulate network conditions between nodes in a Jepsen test using only containers on a single docker host.

Re: Netflix Chaos Monkey Upgraded

#53

Post by niftich » Wed, Oct 19, 2016, 4:49 PM UTC

Thaxll wrote: post #13

ticklemyelmo wrote: post #3

It looks like it's working too well. The site is unreachable for me.

Every time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.

It's NoScript's new-ish defaults at fault.

See https://news.ycombinator.com/item?id=12256720

Re: Netflix Chaos Monkey Upgraded

#54

Post by aaronblohowiak » Wed, Oct 19, 2016, 4:52 PM UTC

yalooze wrote: post #6

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

See my other reply in this thread for our reasoning. https://news.ycombinator.com/item?id=12744567

Re: Netflix Chaos Monkey Upgraded

#55

Post by CorvusCrypto » Wed, Oct 19, 2016, 4:52 PM UTC

Following in line with the jokes, it's like they looked at my company's production environment and said "Let's make that a tool!"

Re: Netflix Chaos Monkey Upgraded

#56

Post by [deleted] » Wed, Oct 19, 2016, 4:54 PM UTC

CorvusCrypto wrote: post #55

Following in line with the jokes, it's like they looked at my company's production environment and said "Let's make that a tool!"

[deleted]

Re: Netflix Chaos Monkey Upgraded

#57

Post by jeremiep » Wed, Oct 19, 2016, 4:57 PM UTC

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

The difference is scale. Only a handful of companies run at Netflix scale.

I wouldn't trust developers to do what Chaos Monkey does at such a scale, no matter how good you think they are.

Re: Netflix Chaos Monkey Upgraded

#58

Post by jaffa214525 » Wed, Oct 19, 2016, 4:58 PM UTC

benhoyt wrote: post #43

jsingleton wrote: post #10

Earlier quoted context omitted.

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)

Immutable infrastructure [1] is preferred. If SSH is detected, you must assume something on the server changed, and has deviated from the baseline.

[1] https://www.oreilly.com/ideas/an-introduction-to-immutable-i...

Re: Netflix Chaos Monkey Upgraded

#59

Post by CorvusCrypto » Wed, Oct 19, 2016, 5:00 PM UTC

iagooar wrote: post #45

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

I don't think even hiring top-notch devs would make this reality completely go away. Tools like this would likely put them in check too. And yeah same response on our internal chat here basically x)

Re: Netflix Chaos Monkey Upgraded

#60

Post by cpeterso » Wed, Oct 19, 2016, 5:00 PM UTC

Beyond external termination services like Chaos Monkey, what are good examples of software that purposely increase internal nondeterminism or failure injection in production?

Go has race detector mode, but it is an optional debug feature with a performance cost. The Linux kernel's jiffy clock starts counting from -5 minutes so drivers must handle clock rollover correctly because it's not a uncommon "once every 48 days" event. Firefox has a chaos debug mode that does things like randomize thread priorities and simulate short socket reads, but that has performance costs.