Live data from Hacker News

The CAP theorem. The Bad, the Bad, & the Ugly

blog.dtornow.com

61–70 of 81 posts

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#61
post #2

The CAP theorem expands to the PACELC theorem. PACELC stands for Partition, Availability, Consistency, Else, Latency, and Consistency. The theorem states that in the event of a network partition, a distributed system must choose between availability and consistency; otherwise, it must choose between latency and consistency. https://en.wikipedia.org/wiki/PACELC_theorem

It seems like people make this really complicated. There will occasionally be network partitions. When there are, a given node can either respond (potentially inconsistently) or not. So you pick some balance between consistency and availability. Latency is really just a proxy for availability - as latency tends towards infinite, availability tends towards zero. Of course you can wait until the network partition is re…

Was going to say, latency doesn't need to be in there.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#62

Earlier quoted context omitted.

Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.

Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific…

I’m not arguing for mainframes here. I’m just saying that when you opt to improve your resiliency and availability by distributing your system, rather than by, say, adding more redundant power supplies and burying your computer in a deeper bunker, you are making your system partitionable, and that means you have to give up either consistency or availability.

Which is odd, when you think about it, since you allegedly made this choice in order to get better availability.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#63

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…

It can be useful to say that everything is a distributed system because it actually matters in a lot of contexts, just not for CAP. There are sorts of networks within a single machine or even a single CPU, and generally "further apart" things have more latency between them. For example, task on talking to another also on vs vs vs all the way off on the GPU... A single failure probably takes down the entire system, but not necessarily.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#64

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…

> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for…

I have seen it happen even within a local dual-redundancy cluster, with machines connected by redundant ethernet links.

Some maintenance person disconnected power to the two servers, then reconnected the power assuming they would reliably recover. The admin instructions were to never do this - always boot up in sequence.

They booted up simultaneously, and the fully redundant network switches took too long to boot fully (Cisco), but started passing internet traffic. After a timeout the servers each assumed they were the master in a degraded cluster, so proceeded to make divergent modifications on the DRBD replicated storage and to serve requests.

I never found out why this happened in spite of the direct ethernet links between the servers which they were supposed to use for synchronisation decisions, but it did.

Recovery required manually comparing changes in files and databases to decide whether to merge or discard.

This problem was avoidable but an adequate fix evidently wasn't in place (mea culpa, limited time and budget).

It did not help that Pacemaker+Corosync was used, before Kubernetes was popular, and Ubuntu Server shipped a very buggy alpha version of Corosync that corrupted itself and crashed often, despite upstream warning it was an unreliable version. I had to manually build a different version of those tools from Red Hat source, because it was too late to change distro. This is one of two reasons I don't recommend Ubuntu Server in professional deployments any more, even though I still use it for my own projects.

Three servers, or two servers and a third special something for arbitration, is a standard solution to this problem.

But it's only useful for a stateful distributed system, like a database or filesystem with some level of multi-master or automatic failover.

There's no need for three nodes or any particular number, for stateless nodes like a web service whose shared state is all calls to a database or filesystem on other nodes.

Technically you don't need three servers. It's enough to have a cheap component or low-cost tiny computer to arbitrate. Even sending commands to the network switches to disable ports (if the switch doesn't behave too strangely, as the Cisco switches did in the above!), or IPMI commands to the other server's BMC. Just about anything can be used, even a high latency, offsite tiny VM, as it isn't needed when the main servers are synchronised.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#65

Earlier quoted context omitted.

They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.

Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.

No matter what, some hardware in that mainframe is making mutexes or some other kind of locking work, and that is a single point of failure. You can't have two of those be authoritative for the same thing, that's the two generals problem.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#66

Earlier quoted context omitted.

Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.

Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific…

Right, if absolutely everything in the mainframe were redundant, that'd be two mainframes connected by a network, so P.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#67

Earlier quoted context omitted.

It seems like people make this really complicated. There will occasionally be network partitions. When there are, a given node can either respond (potentially inconsistently) or not. So you pick some balance between consistency and availability. Latency is really just a proxy for availability - as latency tends towards infinite, availability tends towards zero. Of course you can wait until the network partition is re…

Disagree on latency, for certain classes of applications it really matters. Latency can be caused by a myriad of things, but if that thing is your means for cut over, you haven't failed to respond, you've failed to respond in a timely way. The telco space has strict latency requirements. If you have a failure which results in erratic behaviour on the network due to the database taking time to handle a node failure yo…

Put an SLA on the latency, and it becomes availability.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#68
post #5

In my opinion this dead horse is beat. Anyone working on distributed systems knows that CAP is not useful in isolation. See Eric Brewer's followup article from 2012: https://www.infoq.com/articles/cap-twelve-years-later-how-th...

I had an interview question about it, I tried to cite Kleppmann from the distributed systems book and the interview manager got quite offended.

sounds like you dodged a bullet.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#69

Earlier quoted context omitted.

Fundamental principles often just formally state the obvious. People who want CAP to be profound and useful are missing the point - it just tells you some things that you might want to try to achieve are provably impossible. Just like Newton’s first law tells you things can’t accelerate without a force (duh) and the first law of thermodynamics tells you you can’t get energy out of a perpetual motion machine (duh) and…

Your 'duh's escalated quite quickly, I had a Euclid postulates feelings when reading it - first four are 'duh' and then the fifth one comes swinging. (by the way I think you have your Newton's law order wrong, that's the second one you're referering to)

First law (inertia) basically states that in the absence of a force, no acceleration can happen (sometimes stated as ‘an object in motion will remain in motion, an object at rest will remain at rest’).

Second law gets more specific, and clarifies that the size of the acceleration is proportional to the size of the force.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#70

Earlier quoted context omitted.

It's essential to rule out certain happenings when performing analysis, and acts of God are commonly among them. For example, if you need to account for cosmic rays, you can't prove anything meaningful about software. Redundancy won't get you out of this: you add redundancy, I'll add more cosmic rays. All that does is force a useful analysis into a muddy and probabilistic one. It's actually pretty important to accoun…

That's not what this discussion is about. OP said: > as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. So we are talking in the context of distributed networks. Then: > Mainframes, though, provide consistency and availability by being unpartitionable except through use of a chainsaw. using the…

Your fire is just the chainsaw in the original post.
Post reply on HN