Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix Chaos Monkey Upgraded

techblog.netflix.com

Post reply on HN
Print view View original

31–40 of 89 posts

Re: Netflix Chaos Monkey Upgraded

#31

Post by flgr »

JoachimSchipper wrote: post #23
andrewguenther wrote: post #11

Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?

Yes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/ , because Dan Luu's site is always excellent.) "We rewrote the service for improved maintainability" seems an important part of this blog post.

I wonder if they have a service that automatically detects all those failures and terminates the entire node instead?

Re: Netflix Chaos Monkey Upgraded

#33

Post by notyourwork »

flgr wrote: post #31
JoachimSchipper wrote: post #23

Earlier quoted context omitted.

Yes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/ , because Dan Luu's site is always excellent.) "We rewrote the service for improved maintainability" seems an important part of this blog post.

I wonder if they have a service that automatically detects all those failures and terminates the entire node instead?

I was thinking along these lines as well. Possibly they have other checks and balances in place for less than death situations for servers like high CPU usage and those situations result in another mechanism killing the server which is the same as what this solution provides.

Re: Netflix Chaos Monkey Upgraded

#34

Post by gonzoflip »

fred256 wrote: post #9
dest wrote: post #7

Earlier quoted context omitted.

down : Secure Connection Failed The connection to techblog.netflix.com was interrupted while the page was loading.

It's only available on http, not https.

I am receiving a DNS error when trying to reach the URL

C:\Windows\System32>nslookup http://techblog.netflix.com/ 8.8.8.8

Server: google-public-dns-a.google.com

Address: 8.8.8.8

* google-public-dns-a.google.com can't find http://techblog.netflix.com/: Non-existent domain

Re: Netflix Chaos Monkey Upgraded

#35

Post by haneefmubarak »

jackweirdy wrote: post #16
e1g wrote: post #5

Another useful tool is https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).

I'd love to see this used in the Jepsen tests

AFAIR Jepsen is only meant to test the CAP aspects of a system, not how it behaves otherwise.

Re: Netflix Chaos Monkey Upgraded

#36

Post by takeda »

klapinat0r wrote: post #14
jsingleton wrote: post #8

Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)

How so? The benefits are worth it, and I doubt any CEO will be argue against having fault tolerant code :) You catch bugs, and no one says you can't run Chaos Monkey in staging or a similar environment if it really is a tough sell.

Agreed, it should be hard to explain benefits even to non technical people. It's like doing a fire drill, if you do it frequently when the actual fire happens you will know what to do. Similarly with infrastructure, it might not be good handling rare events, but once these events are not rare you will learn to handle them.

The biggest issue IMO is explaining need to make things more resilient. Actually the technical people (mainly developers) might be the biggest obstacle, because it adds more work for them (with no visible benefit to them, because when application fails it's ops who get woken up).

Re: Netflix Chaos Monkey Upgraded

#37

Post by rglullis »

gonzoflip wrote: post #34
fred256 wrote: post #9

Earlier quoted context omitted.

It's only available on http, not https.

I am receiving a DNS error when trying to reach the URL C:\Windows\System32>nslookup http://techblog.netflix.com/ 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8 * google-public-dns-a.google.com can't find http://techblog.netflix.com/ : Non-existent domain

remove the http:// part and try again?

Re: Netflix Chaos Monkey Upgraded

#38

Post by kgrin »

gonzoflip wrote: post #34
fred256 wrote: post #9

Earlier quoted context omitted.

It's only available on http, not https.

I am receiving a DNS error when trying to reach the URL C:\Windows\System32>nslookup http://techblog.netflix.com/ 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8 * google-public-dns-a.google.com can't find http://techblog.netflix.com/ : Non-existent domain

I'm not on Windows, but I'm pretty sure your problem here is the "http://" - try just:

nslookup techblog.netflix.com 8.8.8.8

Re: Netflix Chaos Monkey Upgraded

#39

Post by yalooze »

scrollaway wrote: post #22
Thaxll wrote: post #13

Earlier quoted context omitted.

Every time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.

"Every time"? If one random guy's complaining about a URL being unreachable and you're already seeing a pattern... is it at all possible the users aren't at fault?

In Thaxll's defense, I've seen this a lot with Netflix blog posts on HN.

Re: Netflix Chaos Monkey Upgraded

#40

Post by mlafeldt »

jsingleton wrote: post #8

Has anyone else deployed a Chaos Monkey in production ? I can imagine it would be a tough sell to the CEO. :)

Yes, we have.

See https://medium.com/production-ready/chaos-monkey-for-fun-and...

As for chaos being a hard sell: https://medium.com/production-ready/chaos-engineering-a-shif...

Post reply on HN