Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Working with the Chaos Monkey

codinghorror.com

Post reply on HN
Print view View original

1–10 of 43 posts

Re: Working with the Chaos Monkey

#3

Post by DanielBMarkham »

I think we're going to be seeing a lot more of Chaos Monkey.

CM is a form of active TDD at the system architecture level. This might evolve into setting up partition tests as a prerequisite to instantiating the deployment model (Translation: before you start putting something on a cloud instance, write code that turns the instance off and on from time to time) This assures that the requirements for survival are baked into the app and not something tacked on later after some public failure like the Amazonocolapse.

I was reading on HN the other day a guy talking about Google. He said he saw engineers pull the wires from dozens of routers handling GBs of data -- all without a hitch. The architecture was baked enough that failure was expected.

Many times failure modes like this are burned into hardware, but that kind of design is a long, long, long way from most people's systems.

Re: Working with the Chaos Monkey

#4

Post by adamc »

Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.

Re: Working with the Chaos Monkey

#5

Post by charper »

adamc wrote: post #4

Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.

This has nothing to do specifically with cloud computing. What is mentioned in netflix's case is but the idea (randomly killing a service) can be used in a more traditional architecture aswel.

Re: Working with the Chaos Monkey

#7

Post by rickdangerous1 »

adamc wrote: post #4

Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.

Chaos monkeys would help with non cloud architectures as well...

Re: Working with the Chaos Monkey

#8

Post by neebz »

Netflix is turning out to be my favourite tech company. Just a week ago, in an extensive interview they mentioned that to provide a consistent interface across so many platforms we ended up porting our own version of webkit. And now the Chaos Monkey. It's amazing how technically sound they are considering they were just an online DVD rental company at the start.

Re: Working with the Chaos Monkey

#9

Post by larrik »

adamc wrote: post #4

Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.

Well, you really have to figure out how much it would cost your site/service to be down for 2 days straight. Or maybe a week (PSN). Would this design pay for itself in preventing that loss?
Post reply on HN