11–15 of 15 posts
Isn't this basically what Google has been doing for years with their DiRT exercises?
2. Netflix has done a great job at publicizing their efforts and open sourcing software that helps you do this kind of testing in a continuous, automated fashion.
So I think the "basically what google has been doing" comment is reductive.
I'm curious though. What does one do if the database goes away? Also, how does one achieve high availability without doubling costs? Is this covered anywhere as a topic area or book or dedicated blog? Quite curious how to get started with something like this.
For your specific question about databases, you generally have clustering to reduce the impact of any one database instance going down, caching of data to guard against temporary db outages/network issues, and sharding of data across multiple databases to reduce the blast radius of any one logical database going away entirely.
>>> Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production. I don't like how they define "Chaos Engineering" as being strictly related to distributed systems.
As a matter of fact, Netflix is running a big distributed system, so that's where they focus their testing efforts. In general, I think it's fair to talk about Chaos Engineering and systems in the general sense, distributed or not.
Isn't this basically what Google has been doing for years with their DiRT exercises?
It is true that Google's Disaster Recovery Testing events are also about breaking things on purpose as a means of preparation. However, those events are typically large-scale, company-wide drills targeting not only critical systems but also business processes involving people.
(They even prevent experts from participating to make sure knowledge is spread across the organization. I recommend reading http://queue.acm.org/detail.cfm?id=2371516 for more.)
As dastbe has pointed out, Chaos Engineering is more about experimenting in a continuous, automated (and hopefully safe) way. Compared to DiRT, experiments are typically smaller in scope, involving fewer people, if any.