Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The CAP theorem. The Bad, the Bad, & the Ugly

blog.dtornow.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#51

Post by IgorPartola »

hodgesrm wrote: post #43
IgorPartola wrote: post #14

I worked on a wide-area distributed system on 2010-2012. CAP and NoSQL was all the rage then (MongoDB is webscale, after all). The approach we took was to have a single database node be the primary and another replicating from it asynchronously. When a node went offline, a third monitoring node made a decision to promote the backup to primary and communicate this decision to all clients. The three nodes were located…

> Lastly, the approach I never explored but was curious about is the idea of the client being responsible for pushing all values to all nodes rather than replication happening in the background. You can client replication at industrial scale by having clients push writes to Kafka and then have multiple, independent systems read and apply them. You still have a SPOF on Kafka of course. Another practical issue is that…

So what I was picturing was more of where the client directly pushes to all the backend nodes simultaneously. That way the client right away knows if it was a success or a failure or a partial success. The whole problem with any of this is that you can have writes be a partial success and what that means cannot be correctly interpreted by the database cluster alone in the general case. Like it’s bad when it happens but is catastrophic or can it be recovered from really depends on what the data is and it should be up to the application to decide that. Instead we are trying to treat databases like it’s always success or failure with maaaaybe some eventual consistency semantics and when a partition happens or a node is simply down the application doesn’t really know if any given piece of data is actually fully durably committed or just partially committed.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#52

Post by rubiquity »

stevefan1999 wrote: post #40

> On the other hand, given a non-permanent partition and the requirements that a request receives a response eventually, that is, in unbounded time, both strong consistency as well as weak consistency can be achieved. Is that why S3 works so well?

The point they're trying to make is that the A in CAP is ambiguous. If your timeout is unbounded and a request completes an hour/day/month/year or more later is the system available?

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#54

Post by kqr »

mbfg wrote: post #53

My understanding of [[The “Pick 2 out of 3” interpretation implies that network partitions are optional, something you can opt-in or opt-out of.]] is you can choose to not have a distributed system.

This is also how I understand it. You choose networked or not networked, and if you choose networked you get a 2-for-1 with partitions.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#55

Post by anyonecancode »

largbae wrote: post #22
jameshart wrote: post #18

Earlier quoted context omitted.

Mainframes have a ton of internal redundancy that enables insanely high avaialability, absent destruction. All ‘availability’ rests on some assumed radius of safety. A nuclear blast in northern Virginia can take out all the availability zones for AWS US-East-1 - that doesn’t invalidate the availability claims of a system that is distributed across three AZs, it just puts an upper limit on it.

"All availability rests on some assumed radius of safety", thank you for that! If this is not a named law could you name it? I have needed this statement in the past but it never came together like that.

I agree it's a good point. A nuclear blast is extrinsic to the system. I think what something like CAP gets us to think about is failures/limitations that are _intrinsic_ to the system. Someone walking around the data center and unplugging all the machines, then flying to the next data center and unplugging them, etc, would also cause all of CAP to fail.

Though, thinking about it more, what it also tells us is that we need to be clear about what "the system" is. There are, after all, some systems that _do_ take nuclear blasts into account, or systems that take "physical breach by a hostile actor" into account.

Less dramatically, we make trade-offs all the time as to what we consider "in" system vs "out", and it's good to be conscious and explicit about them. We see this a lot in UI, especially web UI in terms of things like what browsers to support, whether we care about users' battery life or data cap limits, if we foresee ourselves running the system in a jurisdiction with different regulations around data collection and usage, etc.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#56

Post by valenterry »

filleokus wrote: post #26
valenterry wrote: post #24

Earlier quoted context omitted.

Sure. But the thing is: is there literally not a single component in that mainframe that cannot cause a single point of failure to the whole system? If so, I agree - in that case it would be similar to running two severs in the same AZ. Which provides a certain form of availability. Otherwise I disagree, because as soon as you have a single point of failure, your system is limited by the availability of that specific…

I've been "arguing" AGAINST mainframes on HN in the past, and after that I've seen many really cool presentations about them. One, quite commonly known, is that two separate mainframes can run completely in lock-step if they are less than ≈ 50 km / 30 mi apart. So if one explodes mid instruction it will still continue fine. As for a single system, I don't think there is any single point of failure, you can swap cpu's…

First of all, I'm not arguing against mainframes or anything! I'm just trying to clarify/explain terms and remove confusion. Just to get that out of the way.

> One, quite commonly known, is that two separate mainframes can run completely in lock-step if they are less than ≈ 50 km / 30 mi apart

Since they are 50km apart, they are connected to each other by one or multiple cables. I assume that they are also both connected to the internet (or something else that matters) so that if one burns down the other can take over. Correct so far?

If so, then please answer this questions: how does the system behave if all the cables between the two machines are severed, for an unknown time?

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#57

Post by ithinkso »

jameshart wrote: post #17
zer00eyz wrote: post #8

This conflict comes up everywhere: Newtonian physics/mechanics: good enough in a lot of cases. Einstein accurate, but unnecessary in most cases. In many cases CAP is good enough for us to have the conversation about how the system works. One can then formulate plan for when it doesn't. The fact that it's imperfect at a formal level is academically interesting, but technically irrelevant for a LOT of conversations whe…

Fundamental principles often just formally state the obvious. People who want CAP to be profound and useful are missing the point - it just tells you some things that you might want to try to achieve are provably impossible. Just like Newton’s first law tells you things can’t accelerate without a force (duh) and the first law of thermodynamics tells you you can’t get energy out of a perpetual motion machine (duh) and…

Your 'duh's escalated quite quickly, I had a Euclid postulates feelings when reading it - first four are 'duh' and then the fifth one comes swinging.

(by the way I think you have your Newton's law order wrong, that's the second one you're referering to)

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#58

Post by klabb3 »

j-pb wrote: post #45

There is a theorem called CALM[1] (Consistency as Logical Monotonicity) that shows that systems are eventually consistent iff they are monotonic. I find that reasoning about consistency is much easier from that perspective, for example it immediately gives you an intuition on why individual CRDTs work. 1: https://arxiv.org/abs/1901.01930

Sounds interesting. Is there an intuitive definition of monotonicity for the purposes of distributed systems? I mean, I thought I knew what it means, but I’m not too sure now.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#59

Post by jameshart »

YetAnotherNick wrote: post #32
jameshart wrote: post #7

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…

> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for…

Minimum of three is generally for one of two reasons:

1) to ensure you can deliberately take one offline for maintenance and still have redundancy in case a single node goes offline

2) in some systems an odd number of nodes is needed to ensure no ties in leader elections or decision votes. Three is the smallest odd number that has any redundancy.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#60

Post by valenterry »

samatman wrote: post #49
valenterry wrote: post #16

Earlier quoted context omitted.

They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.

It's essential to rule out certain happenings when performing analysis, and acts of God are commonly among them. For example, if you need to account for cosmic rays, you can't prove anything meaningful about software. Redundancy won't get you out of this: you add redundancy, I'll add more cosmic rays. All that does is force a useful analysis into a muddy and probabilistic one. It's actually pretty important to accoun…

That's not what this discussion is about. OP said:

> as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability.

So we are talking in the context of distributed networks. Then:

> Mainframes, though, provide consistency and availability by being unpartitionable except through use of a chainsaw.

using the words "consistency" and "availability" here is clearly refering to the same words in the sentence before. Hence we are still in the context of distributed networks. And therefore you cannot just rule this out as an "act of god".

Had OP said "A mainframe has high availibility compared to my laptop and for me that is more than enough." then I wouldn't have said anything.

Post reply on HN