Live data from Hacker News

The CAP theorem. The Bad, the Bad, & the Ugly

blog.dtornow.com

31–40 of 81 posts

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#31

Whenever I see questions about CAP as a useful thought exercise in distributed systems, it seems to originate from people trying to defend database systems that don't respond well to the realities of partition inevitably. That is people doing implicit or explicit marketing of some enterprise data landgrab telling you things like they can scale arbitrary SQL joins across distributed tables. Because large data sets do…

Volt is another good example of solving this well. It's consistency guarantees are strong, and offers very high availability through it's replication which commits transactions to replicas simultaneously. In theory it's weaker on the partition side, but in practise with modern hardware and networks it's impact is not felt. It also offers multi data centre replication, with all sites being active.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#32

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…

> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing.

Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for this downtime and not thinking about much more probable ones.

I have asked many who says we need three nodes and three pods for each service minimum, and no one could answer why.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#33
post #2

The CAP theorem expands to the PACELC theorem. PACELC stands for Partition, Availability, Consistency, Else, Latency, and Consistency. The theorem states that in the event of a network partition, a distributed system must choose between availability and consistency; otherwise, it must choose between latency and consistency. https://en.wikipedia.org/wiki/PACELC_theorem

It seems like people make this really complicated. There will occasionally be network partitions. When there are, a given node can either respond (potentially inconsistently) or not. So you pick some balance between consistency and availability. Latency is really just a proxy for availability - as latency tends towards infinite, availability tends towards zero. Of course you can wait until the network partition is re…

Disagree on latency, for certain classes of applications it really matters. Latency can be caused by a myriad of things, but if that thing is your means for cut over, you haven't failed to respond, you've failed to respond in a timely way. The telco space has strict latency requirements. If you have a failure which results in erratic behaviour on the network due to the database taking time to handle a node failure you aren't in a good place, it has a financial impact.

There are databases that can provide high consistency and availability.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#34
post #8

This conflict comes up everywhere: Newtonian physics/mechanics: good enough in a lot of cases. Einstein accurate, but unnecessary in most cases. In many cases CAP is good enough for us to have the conversation about how the system works. One can then formulate plan for when it doesn't. The fact that it's imperfect at a formal level is academically interesting, but technically irrelevant for a LOT of conversations whe…

Fundamental principles often just formally state the obvious. People who want CAP to be profound and useful are missing the point - it just tells you some things that you might want to try to achieve are provably impossible. Just like Newton’s first law tells you things can’t accelerate without a force (duh) and the first law of thermodynamics tells you you can’t get energy out of a perpetual motion machine (duh) and…

[deleted]

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#35
A very common failure mode I see even among experienced senior engineers is to talk about the availability only of their endpoint, not the user experience of the system to meaningfully address user requests. Well-intentioned SLOs around error rate, 99th latency, etc. become meaningless if you don't understand how they affect the various objectives of various production clients.

We generally accept that clients may have to perform retries, definitely jittered and hopefully bounded. Given that's the case, what does it matter whether a single instance takes 5 seconds to come back online after failure (possibly rescheduled on a new node), or multiple instances take the same 5 seconds to recognize leader failure and elect a new leader? Sure, they're unlikely to be identical numbers, but say they're within an order of magnitude and both have error margins so it's a wash.

An architecture astronaut will put forward a design with complex leader election, endpoint discovery, distributed locking, strong consistency, etc. for the latter solution and pat themselves on the back for their expertise and professionalism. They'll waste a lot of time for both dev and SRE as long as the service lives.

A reasonable person should be able to acknowledge that, given either solution will have the client retrying about the same amount of time, the simplest solution that delivers that experience is sufficient from the client's point of view and vastly preferable from a maintainer's or operator's point of view.

Of course not every scenario will be like this. Sometimes startup is unavoidably a lot slower than re-election. It's just rare you see this evaluated in terms of client-facing numbers before a very costly architecture decision is made.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#36

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…

You can have a distributed system that is partially available too. Consider a GPU cluster. A network partition occurs. Both nodes can suspend jobs or refuse to accept jobs that require more than one nodes worth of compute until the partition is healed, while still independently serving jobs that require only one node, while healing all accounting details when reconciliation is possible.

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#37
post #9
post #8

This conflict comes up everywhere: Newtonian physics/mechanics: good enough in a lot of cases. Einstein accurate, but unnecessary in most cases. In many cases CAP is good enough for us to have the conversation about how the system works. One can then formulate plan for when it doesn't. The fact that it's imperfect at a formal level is academically interesting, but technically irrelevant for a LOT of conversations whe…

Agreed. Sure, you can expand it with PACELC, but the important part is that you have something which starts the conversation/thinking about the fact that distributed systems have specific requirements and challenges. And CAP is sufficient for that.

TIL:

> In theoretical computer science, the PACELC theorem is an extension to the CAP theorem. It states that in case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and loss of consistency (C).

* https://en.wikipedia.org/wiki/PACELC_theorem

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#38

The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…

> The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing. Not only that, a weird network partition is highly rare and I have never seen it. In most of the cases, there is only one network and either the server is up and connected to "the" network or is not connected to the network. And I believe engineers have tendency to overengineer for…

[deleted]

Re: The CAP theorem. The Bad, the Bad, & the Ugly

#39
Doesn't strong consistency always imply some form of explicit locking that collides with availability? As in a distributed system it needs to go something like:

1. Get write request to a node

2. Node sends lock on that updated data to rest of nodes

3. After they send back OK the write happens

4. Propagate to rest of nodes

5. After receiving OK on the update from all of the nodes send notification to nodes to lift the lock

So if a partition happens the system fails if it is CA while with e.g. CA it cannot guarantee strong consistency without resolving the split brain (killing of one of the partitions based on e.g. quorum)?

Post reply on HN