If you do not model your software for disaster recovery survivability and routinely test scenarios where you lose a data center, there will certainly be something bad that pops up when it happens in a real-world scenario. Once systems become too complex with too many interacting pieces you need to run real-world DR situations on a schedule to ensure something isn't missed.
Data centers are too reliable
21–27 of 27 posts
Re: Data centers are too reliable
#22I'm reminded of Netflix's Chaos Monkey: http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u...
Ever since reading that last year, whenever a movie I'm watching stops to buffer, I wonder if it could have been the Chaos Monkey.
I think of the chaos monkey more like a baseball umpire: perhaps whenever your movie doesn't stop to buffer, you might thank the chaos monkey.
Re: Data centers are too reliable
#23For 99% of applications, there just isn't budget or a need for software failover. Making data centers continually more reliable serves this 99% well.
Re: Data centers are too reliable
#24Hm. We haven't had the same experience. In the last 3+ years, I recall 1 unplanned outage that could be blamed on "the data center"
We've seen a couple, and I've heard about a couple more. It always seems to be the UPSes, oddly enough.
Re: Data centers are too reliable
#25It used to be, most application code was written to expect everything beneath it (transport, network, data, physical) would work flawlessly. Then people realized, that was just not happening in the real world: - Dual-channel interfaces would still fail (bad line card on the switch). - Redundant Storage frames could suffer catastrophic outages. - Datacenters with dozens of layers of redundancy could still fail due to…
For the network aspects: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...
Re: Data centers are too reliable
#26It used to be, most application code was written to expect everything beneath it (transport, network, data, physical) would work flawlessly. Then people realized, that was just not happening in the real world: - Dual-channel interfaces would still fail (bad line card on the switch). - Redundant Storage frames could suffer catastrophic outages. - Datacenters with dozens of layers of redundancy could still fail due to…
I'd say that most companies aren't at the level where they code for this - they buy COTS software for their business, and much of it is still of the 'put all your eggs in one basket and watch that basket' mentality. I know it works that way at my place of employment. No matter how much I gnash my teeth and wail, the programmers still haven't caught up to current tech.
Re: Data centers are too reliable
#27Earlier quoted context omitted.
Ever since reading that last year, whenever a movie I'm watching stops to buffer, I wonder if it could have been the Chaos Monkey.
I think of the chaos monkey more like a baseball umpire: perhaps whenever your movie doesn't stop to buffer, you might thank the chaos monkey.
Good point. What easy prey I am to biased thinking.