41–50 of 81 posts
This conflict comes up everywhere: Newtonian physics/mechanics: good enough in a lot of cases. Einstein accurate, but unnecessary in most cases. In many cases CAP is good enough for us to have the conversation about how the system works. One can then formulate plan for when it doesn't. The fact that it's imperfect at a formal level is academically interesting, but technically irrelevant for a LOT of conversations whe…
Doesn't strong consistency always imply some form of explicit locking that collides with availability? As in a distributed system it needs to go something like: 1. Get write request to a node 2. Node sends lock on that updated data to rest of nodes 3. After they send back OK the write happens 4. Propagate to rest of nodes 5. After receiving OK on the update from all of the nodes send notification to nodes to lift the…
If you need stronger primitives than atomic read/write (e.g. CAS), then ABD is insufficient and you need consensus in the asynchronous model (although synchronized clocks can allow even CAS to be implemented with ABD + leader election with leases[2]).
[0] https://dl.acm.org/doi/pdf/10.1145/200836.200869
[1] https://www.cl.cam.ac.uk/teaching/2223/ConcDisSys/dist-sys-n...
I worked on a wide-area distributed system on 2010-2012. CAP and NoSQL was all the rage then (MongoDB is webscale, after all). The approach we took was to have a single database node be the primary and another replicating from it asynchronously. When a node went offline, a third monitoring node made a decision to promote the backup to primary and communicate this decision to all clients. The three nodes were located…
You can client replication at industrial scale by having clients push writes to Kafka and then have multiple, independent systems read and apply them. You still have a SPOF on Kafka of course. Another practical issue is that if you take this approach you'll likely need a utility to detect and correct data drift. That's been a feature of the systems I've seen that use this approach successfully.
A very common failure mode I see even among experienced senior engineers is to talk about the availability only of their endpoint, not the user experience of the system to meaningfully address user requests. Well-intentioned SLOs around error rate, 99th latency, etc. become meaningless if you don't understand how they affect the various objectives of various production clients. We generally accept that clients may ha…
I find that reasoning about consistency is much easier from that perspective, for example it immediately gives you an intuition on why individual CRDTs work.
No, it's two out of three. CA systems don't use a network. Consider a database and a work queue which communicates with the database, where they run on the same server. You can achieve consistency and availability in such a system, but only by eliminating partitioning, and there's only the one way to do that: don't run the systems on a network. Nor is this an unrealistic architecture! Far from it, it is in fact the one you should choose if you need both consistency and availability.
The parts of the article about the CAP theorem and its consequences are on fairly solid ground as I see it. But the observation about the CAP conjecture is bootless, the conjecture that CAP is "pick two" holds up to scrutiny. Not that it proves the conjecture, just that all three of the "pick two" options describe meaningful systems.
In my opinion this dead horse is beat. Anyone working on distributed systems knows that CAP is not useful in isolation. See Eric Brewer's followup article from 2012: https://www.infoq.com/articles/cap-twelve-years-later-how-th...
I had an interview question about it, I tried to cite Kleppmann from the distributed systems book and the interview manager got quite offended.
Doesn't strong consistency always imply some form of explicit locking that collides with availability? As in a distributed system it needs to go something like: 1. Get write request to a node 2. Node sends lock on that updated data to rest of nodes 3. After they send back OK the write happens 4. Propagate to rest of nodes 5. After receiving OK on the update from all of the nodes send notification to nodes to lift the…
Invariant confluence, determines whether an application requires coordination for correct execution.
You can tailor the invariants to your requirements making invariant confluence a much better tool than CAP
The ‘network partitions are not optional’ interpretation of CAP is forgetting about the fact that non-distributed-systems are a thing . Sure, as soon as you decided to distribute your system across a network you opted into a world where partition can happen, and you will have to give up consistency or availability. Mainframes, though, provide consistency and availability by being unpartitionable except through use of…
They don't provide availability, because if they go down, they are down. For example, if there is a fire, the whole thing is gone. That's a choice you make.
For example, if you need to account for cosmic rays, you can't prove anything meaningful about software. Redundancy won't get you out of this: you add redundancy, I'll add more cosmic rays.
All that does is force a useful analysis into a muddy and probabilistic one. It's actually pretty important to account for cosmic rays! But you do that on top of an analysis which presumes that the hardware performs correctly. It's not a useful reality to expose to a proof assistant.
To anticipate an objection: yes, you absolutely can rule out network partitions in an analysis, if that's a useful thing to do. For a packet-switched network, it isn't useful. But different network topologies exist, or at least used to, ones where once a circuit is negotiated, you can say useful things about the network without accounting for the kind of equipment failure which might break that circuit. For packet switching, you're going to have a really bad time if you don't account for partitions, because those are expected behavior, one of the properties of the system under consideration.
- CAP came up
- all parties immediately agreed that CA was impossible
I still think it’s a useful model for forcing people to think about their failure modes though. Systems, largely, tend to fall into either CP or AP or be horrendously expensive.
You have to choose your trade offs.
I wish the author had expounded a little further on better options. I’ll read the paper linked, but there’d be more punch with more inline content.