Working with the Chaos Monkey

This is where the distinction is made: "Who in their right mind would willingly choose to work with a Chaos Monkey? Sometimes you don't get a choice; the Chaos Monkey chooses you." But, the final takeaway point of the post blurs that distinction: "the best way to avoid failure is to fail constantly." Stack Exchange didn't "choose" the "best way to avoid failure" -- the problem chose them. Netflix, on the other hand, did make that choice. The article is written as if Stack Exchange followed Netflix's path and chose to fail constantly when it didn't.

"Building things this way strikes me as expensive."

That is a qualitative statement. It implies a value proposition in your head between engineering effort involved in doing CM style disaster prep vs product benefit.

Operationally, not doing some level of CM is like paying for operations with "Lottery Checks". A Lottery check has a payee and a nominal amount but where the amount is actually printed, there is a scratch off box. Sometimes when you scratch it off its for a lot more money than you intended to spend :-).

But it is very hard to talk rationally about "we're spending two engineers here to do nothing but try to randomly break the system and get bugs fixed that would cover for that problem." Because the problem is self inflicted it seems like a waste of money, and there is no guarantee that they will have found and fixed the problem which is going to kill you in the future. However, if you run an experiment enough times, you eventually achieve the solution. Think of it as the Monte Carlo method of systems test. Its a good thing and it helps people sleep at night.

And when the world does go pear shaped like it did with AWS here you may find yourself yawning rather than panicking, and that feels very good indeed.

And 6 months ago, what WAS your perceived probability that an entire AWS location would be down for several days?

A decade or so ago, I heard computer programming described as a very good occupation for a person who had Asperger syndrome or perhaps limited social skills. I also recall reading then that some surveys of programmers working in that era suggested that those programmers were much more introverted than the general population. But I used to notice when I installed new programs on my Microsoft Windows computer, even after installing Windows 95, that sometimes installing one program would disable another program. That made me wonder if maybe social skills are an essential element of good programming skills. Now when software may have to run in the cloud, interacting with other software hosted on other hardware, all attempting to operate synchronously, wouldn't "software social skills" be rather important for any developer to understand?

> And that's why, even though it sounds crazy, the best way to avoid failure is to fail constantly.

This is my biggest concern with things like large nation-states, large banks, large reinsurance companies, large RAIDs, and large nuclear plants: we centralize resources into a larger resource pool in order to reduce the chances of failure, but in doing so we make the eventual failure more severe, and we reduce our experience in coping with it and our ability to estimate its probability. In fact, we may not even be reducing the chances of failure; we may just be fooling ourselves.

Consider the problem of replicating files around a network of servers. Perhaps you have a billion files and 200 single-disk servers with an MTBF of 10 years, and it takes you three days to replace a failed server.

One approach you can use is to pair up the servers into 100 mirrored pairs and put 10 million files on each pair. Now, about 20 servers will fail every year, leaving ten million files un-backed-up for three days. But the chance that the remaining server of that pair will fail during that time is 3/3650 = 0.08%. That will happen about once every 60 years, and so the expected lifetime of the average file on your system is about 6000 years.

So it's likely that your system will hum along for decades without any problems, giving you an enormous sense of confidence in its reliability. But if you divide the files that will be lost once every 60 years (ten million) by the 60 years, you get about 170 thousand files lost per year. The system is fooling you into thinking it's reliable.

Suppose, instead, that you replicate each file onto two servers, but those servers are chosen at random. (Without replacement.) When a server fails (remember, 20 times a year), there's about a one in six chance that another server will fail in the three days before it's replaced. When that happens, every three or four months, a random number of files will be lost --- about 10 million / 200, or about fifty thousand files, for a total data loss of about 170 thousand files a year. You will likely see this as a major problem, and you will undertake efforts to fix it, perhaps by storing each file on three or four servers instead of two.

This is despite the fact that this system loses data at the same average rate as the other one. In effect, instead of having 100 server pairs to store files on, you have 19,900 partition pairs, each partition consisting of 0.5% of a server. By making the independently failing unit much smaller, you've dramatically increased your visibility into its failure rate, and given yourself a lot of experience with coping with its failures.

In this case, more or less by hypothesis, the failure rate is independent of the scale of the thing. That isn't generally the case. If we had a lot of half-megawatt nuclear reactors scattered around the landscape instead of a handful of ten-gigawatt reactors, it's likely that each reactor would receive a lot less human attention to keep it in good repair. When it threatened to melt down, there wouldn't be a team of 200 experienced guys onsite to fight the problem. There would be a lot more shipments of fuel, and therefore a lot more opportunities for shipments of fuel rods to crash or be hijacked. And so on.

But we might still be better off that way, because instead of having to extrapolate nuclear-reactor safety from a total of three meltdowns of production reactors --- TMI, Tchernobyl, and Fukushima --- we'd have dozens, if not hundreds, of smaller accidents. And so we'd know which design elements were most likely to fail in practice, and how to do evacuation and decontamination most effectively. Instead of Tchernobyl having produced a huge cloud of radioactive smoke that killed thousands or tens of thousands of people, perhaps it would have killed 27, like the reactor failure in K-19.

With respect to nation-states, the issue is that strong nation-states are very effective at reducing the peacetime homicide rate, which gives them the appearance of substantially improving safety. Many citizens of strong nation-states in Europe have never lived through a war in their country, leading them to think of deaths by violence as a highly unusual phenomenon. But strong nation-states also create much bigger and more destructive wars. It is not clear that the citizens of, say, Germany are at less risk of death by violence than the citizens of much weaker states such as Micronesia or Brazil, where murder rates are higher.