The Guiness World record for most steps in a Rube Goldberg device was just set at a competition at Purdue University. The device has 244 steps to water a flower! Now, if you saw the Mythbusters' Christmas episode with the Rube Goldberg device, you know it's really hard to make all those steps go right. But in this one, the engineers used a "hammer test": at any point during the operation of the machine, an engineer could tap the side with a hemmer. If it screwed up, that stage was redesigned. http://www.popularmechanics.com/technology/engineering/gonzo... The end result was the most complex machine of its kind, but it runs very reliably.
Working with the Chaos Monkey
21–30 of 43 posts
Re: Working with the Chaos Monkey
#22Earlier quoted context omitted.
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. For sufficiently large MapReduce jobs you can expect one of the compute nodes to fail during the task. If the MapReduce jobs restarted any time this occurred the jobs would never actually complete[1]! As we're allowed to comment on anything public, I'll focus o…
Why are Google using a centralised and closed piece of software? Does it bring many benefits that haven’t been replicated in open alternatives? Or is it just that the cost of switching is high enough to become prohibitive?
I suspect it would be pretty difficult to run a company the size of Google without any closed software (e.g. Oracle Hyperion).
Re: Working with the Chaos Monkey
#23Even linux has a chaos monkey of sorts. http://linux-mm.org/OOM_Killer
Sort of, although in the case of the OOM killer, the goal is actually to kill the "right" process. Of course choosing the "right" process in code turns out to be exceptionally hard over a lot of different workloads.
Re: Working with the Chaos Monkey
#24Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.
A lot of it simply building in redundancy, it has little to do directly with the cloud. And yes, redundancy is always costly, especially if done right. You just need to decide whether your business plan will benefit from it, and what parts are necessary.
The rest of it is making sure your app or site fails gracefully; that is that failure of one part doesn't bring down the whole. That can be expensive to retrofit, but actually should have been designed in from the beginning, as it is a generally accepted part of good design for anything running over a network.
Re: Working with the Chaos Monkey
#25When I first read about the Chaos Monkey, I had assumed it was used on their development/staging environment, but this article implies it is on their production system. Does anyone know which is correct?
I would think you'd have to run it on production for its results to be truly worthwhile.
Re: Working with the Chaos Monkey
#26Here is an erlang version of the chaos monkey:
potential_victim(Minions) ->
fun (Pid) ->
not(pman_process:is_system_process(Pid))
and not lists:member(Pid, Minions)
end.
death_from_above(Minions) ->
Pids = lists:filter(potential_victim(Minions), erlang:processes()),
case Pids of
[] -> none;
_ ->
Victim = lists:nth(random:uniform(length(Pids)), Pids),
Name = pman_process:pinfo(Pid, registered_name),
exit(Victim, kill),
{ok, Victim, Name}
end.
Re: Working with the Chaos Monkey
#27The Chaos Monkey reminds me of some papers I've read about "crash-only software" and "recovery-oriented computing". With this approach, server software is written assuming the only way it would shutdown is a crash, even for scheduled maintenance. The software must be designed to recover safely every time the service is started. Instead of exercising recovery code paths rarely, they are tested every day.
http://www.armandofox.com/geek/past-projects/recovery-orient...
Re: Working with the Chaos Monkey
#28The blog post seems to imply Stack Exchange is working with the Chaos Monkey when it really isn't. They didn't really build a system that randomly shuts down servers or services. The difference is subtle but important.
The point here is that when you know that you're living with the Chaos Monkey, your systems become very fault tolerant. Living with a monkey does that. But even you don't embrace the concept, the Chaos Monkey is likely going to become a uninvited house guest at some point in time. In StackOverflow's case, they bought a mainstream server with a mainstream OS, and discovered that that server came with a monkey. Think o…
Right, I agree that is the main point of the post. However, generalizing Netflix's naming of their fault-inducing system into unexpected failures dilutes the meaning of such name. Jeff's introduction to Netflix's system is what made this post interesting (if you disagree, take the mental exercise and rewrite this post in your mind without mentioning Netflix's system). There's a disconnect between the main point of the post and the most interesting point of the post.
Re: Working with the Chaos Monkey
#29Earlier quoted context omitted.
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. For sufficiently large MapReduce jobs you can expect one of the compute nodes to fail during the task. If the MapReduce jobs restarted any time this occurred the jobs would never actually complete[1]! As we're allowed to comment on anything public, I'll focus o…
Why are Google using a centralised and closed piece of software? Does it bring many benefits that haven’t been replicated in open alternatives? Or is it just that the cost of switching is high enough to become prohibitive?
The short answer is that when Google started, Perforce was the best kid on the block. Once you get as far down the road as Google, it's hard to change that incumbent, even if culture dictates the use of open-source software.
The other problem is that Google just has a whole lot of code. They've got engineers cranking out code all day all over the world. Working at that sort of scale rules out other alternatives (note that Google hired the Subversion guys, and Google isn't using Subversion... this should tell you something).
Re: Working with the Chaos Monkey
#30Here is an erlang version of the chaos monkey: potential_victim(Minions) -> fun (Pid) -> not(pman_process:is_system_process(Pid)) and not lists:member(Pid, Minions) end. death_from_above(Minions) -> Pids = lists:filter(potential_victim(Minions), erlang:processes()), case Pids of [] -> none; _ -> Victim = lists:nth(random:uniform(length(Pids)), Pids), Name = pman_process:pinfo(Pid, registered_name), exit(Victim, kill)…
Yeah, I've been playing around with the idea of Chaos Monkey at the code level, rather than at the systems level, but you can only truly do it with independent actors. I'm hoping to have something to show soon, probably on Akka/Scala.
There is a similarity with mutation testing, but mutation testing is trying to throw things too far up the chain; it wants your program to crash and die so the test fails. Really, we want it the other way: proof that the test would have failed, but the program is still running effectively.
I've worked with runtime repair in the past, which is also sort of similar, but, IMHO, less effective than Erlang-style Let It Crash. [1]