The CAP theorem expands to the PACELC theorem. PACELC stands for Partition, Availability, Consistency, Else, Latency, and Consistency. The theorem states that in the event of a network partition, a distributed system must choose between availability and consistency; otherwise, it must choose between latency and consistency. https://en.wikipedia.org/wiki/PACELC_theorem
It seems like people make this really complicated. There will occasionally be network partitions. When there are, a given node can either respond (potentially inconsistently) or not. So you pick some balance between consistency and availability. Latency is really just a proxy for availability - as latency tends towards infinite, availability tends towards zero. Of course you can wait until the network partition is re…
The CAP theorem. The Bad, the Bad, & the Ugly
61–70 of 81 posts
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#62Earlier quoted context omitted.
Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.
Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific…
Which is odd, when you think about it, since you allegedly made this choice in order to get better availability.
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#63The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#64The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…
> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for…
Some maintenance person disconnected power to the two servers, then reconnected the power assuming they would reliably recover. The admin instructions were to never do this - always boot up in sequence.
They booted up simultaneously, and the fully redundant network switches took too long to boot fully (Cisco), but started passing internet traffic. After a timeout the servers each assumed they were the master in a degraded cluster, so proceeded to make divergent modifications on the DRBD replicated storage and to serve requests.
I never found out why this happened in spite of the direct ethernet links between the servers which they were supposed to use for synchronisation decisions, but it did.
Recovery required manually comparing changes in files and databases to decide whether to merge or discard.
This problem was avoidable but an adequate fix evidently wasn't in place (mea culpa, limited time and budget).
It did not help that Pacemaker+Corosync was used, before Kubernetes was popular, and Ubuntu Server shipped a very buggy alpha version of Corosync that corrupted itself and crashed often, despite upstream warning it was an unreliable version. I had to manually build a different version of those tools from Red Hat source, because it was too late to change distro. This is one of two reasons I don't recommend Ubuntu Server in professional deployments any more, even though I still use it for my own projects.
Three servers, or two servers and a third special something for arbitration, is a standard solution to this problem.
But it's only useful for a stateful distributed system, like a database or filesystem with some level of multi-master or automatic failover.
There's no need for three nodes or any particular number, for stateless nodes like a web service whose shared state is all calls to a database or filesystem on other nodes.
Technically you don't need three servers. It's enough to have a cheap component or low-cost tiny computer to arbitrate. Even sending commands to the network switches to disable ports (if the switch doesn't behave too strangely, as the Cisco switches did in the above!), or IPMI commands to the other server's BMC. Just about anything can be used, even a high latency, offsite tiny VM, as it isn't needed when the main servers are synchronised.
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#65Earlier quoted context omitted.
They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.
Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#66Earlier quoted context omitted.
Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.
Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific…
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#67Earlier quoted context omitted.
It seems like people make this really complicated. There will occasionally be network partitions. When there are, a given node can either respond (potentially inconsistently) or not. So you pick some balance between consistency and availability. Latency is really just a proxy for availability - as latency tends towards infinite, availability tends towards zero. Of course you can wait until the network partition is re…
Disagree on latency, for certain classes of applications it really matters. Latency can be caused by a myriad of things, but if that thing is your means for cut over, you haven't failed to respond, you've failed to respond in a timely way. The telco space has strict latency requirements. If you have a failure which results in erratic behaviour on the network due to the database taking time to handle a node failure yo…
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#68In my opinion this dead horse is beat. Anyone working on distributed systems knows that CAP is not useful in isolation. See Eric Brewer's followup article from 2012: https://www.infoq.com/articles/cap-twelve-years-later-how-th...
I had an interview question about it, I tried to cite Kleppmann from the distributed systems book and the interview manager got quite offended.
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#69Earlier quoted context omitted.
Fundamental principles often just formally state the obvious. People who want CAP to be profound and useful are missing the point - it just tells you some things that you might want to try to achieve are provably impossible. Just like Newton’s first law tells you things can’t accelerate without a force (duh) and the first law of thermodynamics tells you you can’t get energy out of a perpetual motion machine (duh) and…
Your 'duh's escalated quite quickly, I had a Euclid postulates feelings when reading it - first four are 'duh' and then the fifth one comes swinging. (by the way I think you have your Newton's law order wrong, that's the second one you're referering to)
Second law gets more specific, and clarifies that the size of the acceleration is proportional to the size of the force.
Re: The CAP theorem. The Bad, the Bad, & the Ugly
#70Earlier quoted context omitted.
It's essential to rule out certain happenings when performing analysis, and acts of God are commonly among them. For example, if you need to account for cosmic rays, you can't prove anything meaningful about software. Redundancy won't get you out of this: you add redundancy, I'll add more cosmic rays. All that does is force a useful analysis into a muddy and probabilistic one. It's actually pretty important to accoun…
That's not what this discussion is about. OP said: > as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. So we are talking in the context of distributed networks. Then: > Mainframes, though, provide consistency and availability by being unpartitionable except through use of a chainsaw. using the…