The CAP theorem. The Bad, the Bad, & the Ugly

My first thought was The Structure of Scientific Revolution. CAP was a very useful theory on top of which years of research were done until the paradigm was broken and replaced. That's how progress works. It doesn't mean CAP was a scam it was our very good first draft at formalizing distributed database theory. And the fact that we still talk about it is proof of that.

I worked on a wide-area distributed system on 2010-2012. CAP and NoSQL was all the rage then (MongoDB is webscale, after all).

The approach we took was to have a single database node be the primary and another replicating from it asynchronously. When a node went offline, a third monitoring node made a decision to promote the backup to primary and communicate this decision to all clients. The three nodes were located in three different cities and as such the possibility of all three having issues at once was reduced. By doing this, we could literally pull the plug on the primary node and within 30 seconds the clients were talking to the backup-now-primary. If the monitoring node failed, of course we had no way of automatically switching database nodes but again it had to be a major issue for a node in Seattle to get cut off at the same time as the nodes in Dallas or DC. This worked because of the kinds of data we processed so that our 30 second failover time was acceptable AND a small window of lost writes was also OK.

I think another interesting approach is to have your nodes explicitly communicate write status when responding. Basically when you write(node1, key, val) and expect node1 to normally propagate the write to node2, in the case of a network partition node1 would respond with ack(key, nodes_written=[node1]) explicitly excluding node2 from nodes_written if it was unable to push the write to node2 in a timely fashion. Similarly, read(node1, key) could have node1 return resp(key, val, inconsistent=true) or again spelling out which nodes do or do not know this value.

This would give the application a way to decide if the write should be considered successful or not based on what key represents. For example, updating the current position of a fast moving object that sends updates all the time could easily lose a write or two without it being a problem for the user, but a financial transaction could not.

Lastly, the approach I never explored but was curious about is the idea of the client being responsible for pushing all values to all nodes rather than replication happening in the background. This would effectively allow the write to happen simultaneously and also know if the write was a fail, a partial success, or a full success. Then a read could be done from just one node, but before returning it would poll other nodes and return the value that is consistent amongst the majority of the nodes or a failure if it could not reach any of them.

They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.

Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety.

A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.

Datacenter fires cause downtime no matter what consistency model you choose.

Also, mainframes are pretty big, multi-processor machines. They have their problems, but having all those components give out all at once? I'm sure it happens, but I've never heard of it.