argh! so why was the server crashing? you can't leave me in such suspense....!
http://serverfault.com/questions/104791/windows-server-2008-...
Broadcom networking card and Windows Server 2008
Working with the Chaos Monkey
11–20 of 43 posts
Re: Working with the Chaos Monkey
#12When I first read about the Chaos Monkey, I had assumed it was used on their development/staging environment, but this article implies it is on their production system. Does anyone know which is correct?
Re: Working with the Chaos Monkey
#13The blog post seems to imply Stack Exchange is working with the Chaos Monkey when it really isn't. They didn't really build a system that randomly shuts down servers or services. The difference is subtle but important.
Re: Working with the Chaos Monkey
#14The blog post seems to imply Stack Exchange is working with the Chaos Monkey when it really isn't. They didn't really build a system that randomly shuts down servers or services. The difference is subtle but important.
I think the post makes the distinction very clearly. First we hear about Netflix doing this surprising thing on purpose. Then we hear about the Stack Exchange guys pulling out their hair to fix a recurring problem. But "even in our time of greatest frustration, I realized that there was a positive side to all this." Where's the ambiguity?
Re: Working with the Chaos Monkey
#15Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.
Well, you really have to figure out how much it would cost your site/service to be down for 2 days straight. Or maybe a week (PSN). Would this design pay for itself in preventing that loss?
...and then multiply that cost by the probability that this will happen to find the expected payoff. If the work costs substantially more than the payoff, don't bother. If substantially less, you're negligent if you don't do it. (The Learned Hand Rule http://aler.oxfordjournals.org/content/7/2/523.full)
Re: Working with the Chaos Monkey
#16I think we're going to be seeing a lot more of Chaos Monkey. CM is a form of active TDD at the system architecture level. This might evolve into setting up partition tests as a prerequisite to instantiating the deployment model (Translation: before you start putting something on a cloud instance, write code that turns the instance off and on from time to time) This assures that the requirements for survival are baked…
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. For sufficiently large MapReduce jobs you can expect one of the compute nodes to fail during the task. If the MapReduce jobs restarted any time this occurred the jobs would never actually complete[1]!
As we're allowed to comment on anything public, I'll focus on a paper about disaster recovery at Google which focuses on the Perforce version control system[2]. As Perforce is centralized and proprietary it even raises a few novel issues. As they can't modify the code themselves it's in fact one of the few vertically scaled pieces of software at Google (Perforce instances run on machines with 256GB of RAM[3]).
Of particular interest to me is the Annual Disaster Recovery Test that Google runs. They assume that the admins/engineers at Mountain View are entirely unavailable and that the fail-overs happen with no advance notice. The idea is that during an actual disaster your staff won't have time to answer queries as to which folder that documentation was in or the order that commands need to be run.
[1] - This is in one of the official Google MapReduce papers, I'll try and hunt it down
[2] - http://www.perforce.com/perforce/conferences/us/2009/Present...
[3] - http://www.perforce.com/perforce/conferences/us/2009/Present...
Re: Working with the Chaos Monkey
#17The blog post seems to imply Stack Exchange is working with the Chaos Monkey when it really isn't. They didn't really build a system that randomly shuts down servers or services. The difference is subtle but important.
The point here is that when you know that you're living with the Chaos Monkey, your systems become very fault tolerant. Living with a monkey does that.
But even you don't embrace the concept, the Chaos Monkey is likely going to become a uninvited house guest at some point in time. In StackOverflow's case, they bought a mainstream server with a mainstream OS, and discovered that that server came with a monkey.
Think of it another way. My brother and sister in law never worried about the failure characteristics of dinner plates, so they had lots of nice stuff. Then they had a baby. All of the sudden, falling plates and glasses became something that they had to think about.
Re: Working with the Chaos Monkey
#18Even linux has a chaos monkey of sorts. http://linux-mm.org/OOM_Killer
Re: Working with the Chaos Monkey
#19I think we're going to be seeing a lot more of Chaos Monkey. CM is a form of active TDD at the system architecture level. This might evolve into setting up partition tests as a prerequisite to instantiating the deployment model (Translation: before you start putting something on a cloud instance, write code that turns the instance off and on from time to time) This assures that the requirements for survival are baked…
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. For sufficiently large MapReduce jobs you can expect one of the compute nodes to fail during the task. If the MapReduce jobs restarted any time this occurred the jobs would never actually complete[1]! As we're allowed to comment on anything public, I'll focus o…
Why are Google using a centralised and closed piece of software? Does it bring many benefits that haven’t been replicated in open alternatives? Or is it just that the cost of switching is high enough to become prohibitive?
Re: Working with the Chaos Monkey
#20Earlier quoted context omitted.
As you say Google were one of the pioneers of the Chaos Monkey concept; they simply run at a scale where the Chaos Monkey occurs through normal failure rates. For sufficiently large MapReduce jobs you can expect one of the compute nodes to fail during the task. If the MapReduce jobs restarted any time this occurred the jobs would never actually complete[1]! As we're allowed to comment on anything public, I'll focus o…
Why are Google using a centralised and closed piece of software? Does it bring many benefits that haven’t been replicated in open alternatives? Or is it just that the cost of switching is high enough to become prohibitive?
Perforce was arguably the best VCS for large groups when Google first started over a decade ago. It's difficult to move all that code and history and the accumulated tools to a new system.