Live data from Hacker News

Making CRDTs Byzantine Fault Tolerant [pdf]

martin.kleppmann.com

31–40 of 41 posts

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#31
post #24

Earlier quoted context omitted.

Consensus is only necessary when one needs to decide between conflicting choices, and CRDTs are by definition conflict-free. Nodes only need to verify that any update to the CRDT is in fact a legal operation on the data structure. Essentially, the paper details how to ensure two properties: (1) All nodes will eventually receive all legal updates, and (2) no node will accept a malformed update, (and crafting undetecta…

Could this be used to prevent double spends by marking such a spend as 'an illegal update'? I imagine not, but I wonder what limitation of CRDTs would prevent this.

spends don’t commute: if two transactions both consume $1, and the senders account only contains $1, there’s no way to apply both transactions and end up in a valid state.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#32
post #25

Earlier quoted context omitted.

Did he say anything on the topic of the book?

Not really, in his Patreon page he mentions that one of his plans is to "Write another book to complement Designing Data-Intensive Applications"

Designing Compute-intensive Applications?

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#33
post #26
post #7

> The 3f + 1 assumption means these protocols cannot be deployed in open peer-to-peer systems, since they would be vulnerable to Sybil attacks. In contrast, my approach makes no assumption about the number of Byzantine nodes. I'm confused - probably because I haven't finished reading the paper. 1. Sybil-proof-ness requires a CA [1]. It's orthogonal to whether or not a protocol is BFT. Specifically, the classical BFT…

Sybil proofness does not require a CA. The abstract of your link only states that CAs can be a solution. An alternative approach would be to have the system function as long as one member remains non-sybil.

The paper is simple and readable enough for me to not have to comment. Anyway, The abstract itself says:

> This paper shows that, without a logically centralized authority, Sybil attacks are always possible except under extreme and unrealistic assumptions of resource parity and coordination among entities.

And the proof is a few pages later in a few Lemmas. Note that this paper is where Sybil attacks actually come from, and you can see almost all DHT security papers (eg. Castro’s secure routing paper [1]) assume a CA.

[1] section 3.2 of https://www.cs.rice.edu/~dwallach/pub/osdi2002.pdf

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#34
post #5

I recently finished a 2nd reading of DDIA and while listening to a podcast featuring an interview with Martin Kleppmann, I got so thrilled when he mentioned that he plans to release another book in the coming years. DDIA is so good that I feel the same kind of anticipation I have as when waiting for a next in a series fantasy book to take me back to a parallel world with characters I've come to love and miss like one…

Does this book give ways to deal with the following: a mobile app that works offline (local replica), and then when it's online, syncs (through a server) with other devices' local replicas? I don't even know what to call this situation. I'm sure CRDTs could help with this.

I guess the book is more of a very well structured fast path (compared to learning from blogs) to give you a fundamental understanding of storing, transferring data. You won’t find the exact answer to your question, but reading this book enlightens you to read other things and helps you better to understand what solution to use in what case.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#35

The idea of DAG-embedded CRDTs is far from new and was introduced here: https://arxiv.org/abs/2004.00107 (I'm among the authors) Unfortunately, the verification that the author proposes (not accepting new updates until the dag below is verified) will need a lot of caveats for real world usage. Currently we use these CRDTs for a key value database of 40M+ keys in a deployment of ipfs-cluster, which uses https://github…

Martin here. I cited your paper in the related work section. It's a good start, but it does not cover everything that's required to achieve BFT — in particular, the issue that an operation may be valid in some contexts and invalid in others, and all correct nodes need to agree on whether an operation is valid or not. If you have more details on the caveats you have discovered, it would be great if you could write the…

Hi Martin,

I now had time to read more closely. Thank you for citing us!

In our paper and original approach/algo (iirc), CRDT updates are applied from the bottom of the dag to the heads (older to recent). This is also what you propose although we don't discuss how this helps wrt BFT. I think it is a very good point to highlight that hash-graph based CRDTs (or Merkle-CRDTs as well call them) can provide this property when processed "in order".

The caveat is that, in practice we found this quite impractical: when receiving an update with unknown descendants, you'd have to choose whether to hold on it to apply it at a later point when the missing sub-dag is received (which may never happen), or to drop it and rely on re-transmission at a later point. The first opens the door to abuse by a bad actor because you spend memory holding on to un-applied updates. The latter causes additional work as updates could pile up over an missing one, thus incurring re-transmission costs for the missing and all the later elements.

This also means that peers pubsub systems should probably decide whether to re-broadcast an update based on whether is valid or not per the CRDT operation it contains, which can have bad side-effects: a network missing an update due to a network issue may block the broadcasting of later any later updates even if they are correct, thus worsening the delivery for everyone and forcing the network to issue more re-transmissions.

And as a result, a peer should also decide whether to issue/broadcast updates based on whether the previous update has at least been received by a non Byzantine replica, as otherwise updates built on top will not be accepted by the network until re-transmission. That makes another potential bottleneck.

In our implementation, we found more practical to process updates immediately as they are received, and then process descendants until we reach the "known" part of the DAG. This means every update can be broadcast, gossiped around, and will be potentially applied without doing any waiting for parents, and occasional loss of an update does not block the processing of new updates before re-transmission. If an update has descendants that do not exist then it can be considered a DAG with a different "root" and does not have many consequences in terms of convergence . Note that here we are talking about DAGs with depth of 100k+ items, where sometimes there are 200 heads and processing every update may take a few seconds. We need to avoid blocking a replica as much as possible and get as much work done as possible asap.

I think some CRDTs can get away with this (in our case, CRDT-add-only-sets, using the hashes as IDs) and be byzantine-failure resistant (things would converge). In the paper you mention some examples where CRDT-rules can be abused more easily, so I'm guessing it is more difficult other CRDT types. Ensuring that IDs are unique is one of the main advantages of using hash trees.

In general, I think an attacker can usually find ways to screw up with a non-permissioned CRDT system without breaking convergence (i.e by submitting many valid updates). Your approach makes however a very good point wrt to misbehaving nodes which are not necessarily malicious.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#36

The idea of DAG-embedded CRDTs is far from new and was introduced here: https://arxiv.org/abs/2004.00107 (I'm among the authors) Unfortunately, the verification that the author proposes (not accepting new updates until the dag below is verified) will need a lot of caveats for real world usage. Currently we use these CRDTs for a key value database of 40M+ keys in a deployment of ipfs-cluster, which uses https://github…

Protocol looks computationally expensive to me. I think the dag size could get out of hand if an offline but otherwise correct node comes back online. What caveats do you have in mind?

DAG size gets out of hand when many operations happen (akin to how a blockchain grows indefinitely).

Nodes could however rely on some form of "consensus" to compact/reset the graph at intervals. But then you have invited a consensus into the party which is the opposite of what CRDTs want to have.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#37
post #24

Earlier quoted context omitted.

Could this be used to prevent double spends by marking such a spend as 'an illegal update'? I imagine not, but I wonder what limitation of CRDTs would prevent this.

spends don’t commute: if two transactions both consume $1, and the senders account only contains $1, there’s no way to apply both transactions and end up in a valid state.

That makes sense. Thanks.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#38
> Further work is required to demonstrate whether the techniques presented here are indeed effective in the context of particular CRDT algorithms, to prove their correctness in the face of Byzantine nodes, and to measure the performance impact of Byzantine fault tolerance.

This paper is super interesting, but it'd be even more interesting when we see this future work is completed.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#39
post #34

Earlier quoted context omitted.

Does this book give ways to deal with the following: a mobile app that works offline (local replica), and then when it's online, syncs (through a server) with other devices' local replicas? I don't even know what to call this situation. I'm sure CRDTs could help with this.

I guess the book is more of a very well structured fast path (compared to learning from blogs) to give you a fundamental understanding of storing, transferring data. You won’t find the exact answer to your question, but reading this book enlightens you to read other things and helps you better to understand what solution to use in what case.

I agree, that it's more of a taxonomy and reasoning guide for distributed systems, but the situation in question, regarding long replication lag for offline devices is introduced as an example (through a calendar app) when discussing multi-leader replication.

Re: Making CRDTs Byzantine Fault Tolerant [pdf]

#40
post #25

Earlier quoted context omitted.

Not really, in his Patreon page he mentions that one of his plans is to "Write another book to complement Designing Data-Intensive Applications"

Designing Compute-intensive Applications?

Reminds me of "These two books contain the sum total of all human knowledge": https://imgur.com/wf53LHK
Post reply on HN