Dat – Distributed Dataset Synchronization and Versioning
1–10 of 40 posts
Re: Dat – Distributed Dataset Synchronization and Versioning
#2We use Dat in Beaker[1] to host sites and files from the user's device. It's a pretty interesting protocol. It's developed by Code for Science[2], a 501(c)(3) led by Max Ogden[3] and with protocol dev led by Mafintosh[4]; their mission is to help with archival of science and civic data.
Some interesting properties:
1. It uses a BitTorrent-style of swarm, but primarily to sync signed append-only logs, which are in fact flattened Merkle Trees (similar to Certificate Transparency). The Dat archives are addressed by public keys. The tree is used to enforce the append-only constraint by making it easy to detect if the history has been changed by the author.
2. The "Secret Sharing" feature. The public key of a Dat archive is hashed before querying or announcing on the discovery network, and then the traffic is encrypted using the public key as a symmetric key. This has the effect of hiding the content from the network, and thus making the public key of a Dat a "read capability": you have to know the key to access its files.
There's a reference implementation in JS available at https://github.com/datproject/dat-node, and a fair number of tools being built around it.
Re: Dat – Distributed Dataset Synchronization and Versioning
#3We use Dat in Beaker[1] to host sites and files from the user's device. It's a pretty interesting protocol. It's developed by Code for Science[2], a 501(c)(3) led by Max Ogden[3] and with protocol dev led by Mafintosh[4]; their mission is to help with archival of science and civic data. Some interesting properties: 1. It uses a BitTorrent-style of swarm, but primarily to sync signed append-only logs, which are in fac…
I wish Beaker had picked a different name. It collides with the Beaker Notebook, a Jupyter alternative that unfortunately never seemed to gain traction but had some really killer features that Jupyter has yet to pick up, especially the ability to mix Python, R, and Julia cells in the same notebook.
Re: Dat – Distributed Dataset Synchronization and Versioning
#4We use Dat in Beaker[1] to host sites and files from the user's device. It's a pretty interesting protocol. It's developed by Code for Science[2], a 501(c)(3) led by Max Ogden[3] and with protocol dev led by Mafintosh[4]; their mission is to help with archival of science and civic data. Some interesting properties: 1. It uses a BitTorrent-style of swarm, but primarily to sync signed append-only logs, which are in fac…
I wish Beaker had picked a different name. It collides with the Beaker Notebook, a Jupyter alternative that unfortunately never seemed to gain traction but had some really killer features that Jupyter has yet to pick up, especially the ability to mix Python, R, and Julia cells in the same notebook.
Yeah I regret that. We'll consider a rename at some point. Here's their project: http://beakernotebook.com/
Re: Dat – Distributed Dataset Synchronization and Versioning
#5Wasn't dat originally going to be a part of ipfs or was it the browser? What are the reasons for dat vs ipfs?
Re: Dat – Distributed Dataset Synchronization and Versioning
#6Wasn't dat originally going to be a part of ipfs or was it the browser? What are the reasons for dat vs ipfs?
Contrary to IPFS dat is more focused on what can be achieved right now while sacrificing true decentralisation, and is more opinonated with providing in its use case with providing things like versioning.
Re: Dat – Distributed Dataset Synchronization and Versioning
#7Wasn't dat originally going to be a part of ipfs or was it the browser? What are the reasons for dat vs ipfs?
They're pretty similar. Beaker supported both once, which may be what you're thinking of. This is what guided our decision: https://beakerbrowser.com/docs/inside-beaker/other-technolog...
The two features I list above, the append-only histories and secret-sharing, are unique to Dat. And, for us, the URL spec was a big deal.
Re: Dat – Distributed Dataset Synchronization and Versioning
#8Wasn't dat originally going to be a part of ipfs or was it the browser? What are the reasons for dat vs ipfs?
Contrary to IPFS dat is more focused on what can be achieved right now while sacrificing true decentralisation, and is more opinonated with providing in its use case with providing things like versioning.
What do you think it sacrifices? IPFS is more focused on static blob addressing while Dat focuses on data sources, and in that sense, there’s a single authority over a dataset. But I see that as a positive, since mutability is pretty valuable.
TBH, I'm not sure how a site can work without mutability.
Re: Dat – Distributed Dataset Synchronization and Versioning
#9Earlier quoted context omitted.
Contrary to IPFS dat is more focused on what can be achieved right now while sacrificing true decentralisation, and is more opinonated with providing in its use case with providing things like versioning.
What do you think it sacrifices? IPFS is more focused on static blob addressing while Dat focuses on data sources, and in that sense, there’s a single authority over a dataset. But I see that as a positive, since mutability is pretty valuable. TBH, I'm not sure how a site can work without mutability.
a dynamic site can't work without mutability, ipfs can't deal with mutability, so ipfs comes with ipns, which allows you to statically reference content that might change.
Re: Dat – Distributed Dataset Synchronization and Versioning
#10Max Ogden and Mafintosh are incredibly talented and productive. Awe-inspiring to see the things they make.