Live data from Hacker News

Building a BFT JSON CRDT

jzhao.xyz

21–30 of 38 posts

Re: Building a BFT JSON CRDT

#21

For people like me: BFT - Byzantine Fault Tolerant [0] CRDT - Conflict-free Replicated Data Type [1] [0] https://en.wikipedia.org/wiki/Byzantine_fault [1] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

RGA/CT - Replicated Growable Array/Causal Tree

I don't like unexplained acronyms/initialisms.

Re: Building a BFT JSON CRDT

#22
> I write this blog post mostly as a note to my past self, distilling a lot of what I’ve learned since into a blog post I wish I had read before going in

The best kind of blog post!

Re: Building a BFT JSON CRDT

#23

It's the Byzantine Fault Tolerant part of this that is particularly innovative and based on Kleppmanns most recent work. I believe it solves the issue of either malicious actors in the networks modifying others transactions, spoofing them, or the messages being modified by third parties ("outside" the network) who have MITM the connection. These are really great problems to solve. However, when I was experimenting wi…

I think this is the schema system you're asking for: https://www.hyperhyperspace.org/ (specifically, the data representation part)

This is a cool idea, but I didn't find any examples of max length constraints or other normalization rules in the source code I reviewed. Maybe there's something in there.

Here's some source code for an early, work-in-progress Wiki CRDT: https://github.com/hyperhyperspace/wiki-collab/blob/master/s...

Page in the Wiki. Note that data types have a validate method that returns true or false; maybe if false, they're just dropped from the UI? Not sure how the method is used. https://github.com/hyperhyperspace/wiki-collab/blob/master/s...

I haven't found the underlying text or rich text CRDT implementation yet.

Re: Building a BFT JSON CRDT

#24

It's the Byzantine Fault Tolerant part of this that is particularly innovative and based on Kleppmanns most recent work. I believe it solves the issue of either malicious actors in the networks modifying others transactions, spoofing them, or the messages being modified by third parties ("outside" the network) who have MITM the connection. These are really great problems to solve. However, when I was experimenting wi…

Interesting, yeah access-control is kinda open problem with Yjs. Regarding ProseMirror and rich-text documents, you can mess up documents in other ways as well. Eg deploy a faulty command with a transaction that inserts nodes with invalid children (can be prevented though by using createChecked). Or just changing your schema in an incompatible way with the previous version. So you kinda have to deal with possible malformed documents either way.

Havent had documents corrupted by Yjs allowing changes that are not parseable by schema though - has this happened to you?

And about the schema layer on top of Yjs, you possibly could inspect every update and apply some validation rules. Arent all operations just inserts, updates or deletes of nodes? You can at least rollback to previous version as you flush the updates to the doc in the db. Not ideal though.

Re: Building a BFT JSON CRDT

#25

It's the Byzantine Fault Tolerant part of this that is particularly innovative and based on Kleppmanns most recent work. I believe it solves the issue of either malicious actors in the networks modifying others transactions, spoofing them, or the messages being modified by third parties ("outside" the network) who have MITM the connection. These are really great problems to solve. However, when I was experimenting wi…

When I was working on this problem with Fluid Framework, we did a few interesting experiments with "owned objects" and centralized ACL objects. I believe the team primarily implemented centralized ACL because that implementation works for many enterprise use cases.

With a centralized schema provider, you run a connected node on a trusted server and reject changes that are out of schema or should not be accessed by a user.

An owned object is an object where a user (or user group that votes via quorum) that owns the object can veto changes to the object. The changes are temporarily applied until accepted by the owners. I haven't dug deep enough into this BFT implementation to know how our model would map to this model.

Re: Building a BFT JSON CRDT

#26
post #23

Earlier quoted context omitted.

I think this is the schema system you're asking for: https://www.hyperhyperspace.org/ (specifically, the data representation part)

This is a cool idea, but I didn't find any examples of max length constraints or other normalization rules in the source code I reviewed. Maybe there's something in there. Here's some source code for an early, work-in-progress Wiki CRDT: https://github.com/hyperhyperspace/wiki-collab/blob/master/s... Page in the Wiki. Note that data types have a validate method that returns true or false; maybe if false, they're just…

Hey! Author of the post responding here :) Unfortunately max/min-length is a global invariant and not one that can be reinforced by CRDTs without coordination

bloom-lang.net is a really cool project working on trying to figure out what types of program state actually require coordination at compile time

Re: Building a BFT JSON CRDT

#28
post #24

It's the Byzantine Fault Tolerant part of this that is particularly innovative and based on Kleppmanns most recent work. I believe it solves the issue of either malicious actors in the networks modifying others transactions, spoofing them, or the messages being modified by third parties ("outside" the network) who have MITM the connection. These are really great problems to solve. However, when I was experimenting wi…

Interesting, yeah access-control is kinda open problem with Yjs. Regarding ProseMirror and rich-text documents, you can mess up documents in other ways as well. Eg deploy a faulty command with a transaction that inserts nodes with invalid children (can be prevented though by using createChecked). Or just changing your schema in an incompatible way with the previous version. So you kinda have to deal with possible mal…

No, I haven't seen a corrupt "unparceable" document. It's more issues around documents that don't comply with the schema.

Say for example you have a node that can contain a single optional . If two users concurrently add a caption, then merge their changes, the Yjs document will contain both. It has no concept of what the valid structure is. When this is loaded into the ProseMirror the second caption will be dropped.

Re: Building a BFT JSON CRDT

#29
post #24

Earlier quoted context omitted.

Interesting, yeah access-control is kinda open problem with Yjs. Regarding ProseMirror and rich-text documents, you can mess up documents in other ways as well. Eg deploy a faulty command with a transaction that inserts nodes with invalid children (can be prevented though by using createChecked). Or just changing your schema in an incompatible way with the previous version. So you kinda have to deal with possible mal…

No, I haven't seen a corrupt "unparceable" document. It's more issues around documents that don't comply with the schema. Say for example you have a node that can contain a single optional . If two users concurrently add a caption, then merge their changes, the Yjs document will contain both. It has no concept of what the valid structure is. When this is loaded into the ProseMirror the second caption will be dropped.

Hmm hadn't thought about that. Yeah Yjs should use more granular steps instead of just replacing the whole doc each time remote update is received.

Re: Building a BFT JSON CRDT

#30

It's the Byzantine Fault Tolerant part of this that is particularly innovative and based on Kleppmanns most recent work. I believe it solves the issue of either malicious actors in the networks modifying others transactions, spoofing them, or the messages being modified by third parties ("outside" the network) who have MITM the connection. These are really great problems to solve. However, when I was experimenting wi…

When I was working on this problem with Fluid Framework, we did a few interesting experiments with "owned objects" and centralized ACL objects. I believe the team primarily implemented centralized ACL because that implementation works for many enterprise use cases. With a centralized schema provider, you run a connected node on a trusted server and reject changes that are out of schema or should not be accessed by a…

How does that compare to Google’s Zanzibar?
Post reply on HN