Earlier quoted context omitted.
What do you think it sacrifices? IPFS is more focused on static blob addressing while Dat focuses on data sources, and in that sense, there’s a single authority over a dataset. But I see that as a positive, since mutability is pretty valuable. TBH, I'm not sure how a site can work without mutability.
a dynamic site can't work without mutability, ipfs can't deal with mutability, so ipfs comes with ipns, which allows you to statically reference content that might change.
Dat – Distributed Dataset Synchronization and Versioning
11–20 of 40 posts
Re: Dat – Distributed Dataset Synchronization and Versioning
#12Re: Dat – Distributed Dataset Synchronization and Versioning
#13Wasn't dat originally going to be a part of ipfs or was it the browser? What are the reasons for dat vs ipfs?
They're pretty similar. Beaker supported both once, which may be what you're thinking of. This is what guided our decision: https://beakerbrowser.com/docs/inside-beaker/other-technolog... The two features I list above, the append-only histories and secret-sharing, are unique to Dat. And, for us, the URL spec was a big deal.
IPFS still has the long-term goal of path addressing (NURI), we just hadn't yet completely figured out what the upgrade path should look like. The discussion linked above is turning into a spec and into actions, i.e. IPFS will be trying to get as much as possible of Electron's protocol API [2] into WebExtensions, [3] and make us of that in the browser addons. [4]
[1] https://github.com/ipfs/specs/pull/152#issuecomment-28462886...
[2] https://electron.atom.io/docs/api/protocol/
Re: Dat – Distributed Dataset Synchronization and Versioning
#14Earlier quoted context omitted.
They're pretty similar. Beaker supported both once, which may be what you're thinking of. This is what guided our decision: https://beakerbrowser.com/docs/inside-beaker/other-technolog... The two features I list above, the append-only histories and secret-sharing, are unique to Dat. And, for us, the URL spec was a big deal.
Actually there's been some movement regarding URLs :) You can check out the reasoning and plan here: [1] IPFS still has the long-term goal of path addressing (NURI), we just hadn't yet completely figured out what the upgrade path should look like. The discussion linked above is turning into a spec and into actions, i.e. IPFS will be trying to get as much as possible of Electron's protocol API [2] into WebExtensions,…
Here's why I think you're shooting yourself in the foot with the NURIs, though.
1) You're focusing on syntax. The "there is no domain" premise works just as fine with your stage two of `ipfs://{hash}`, so dismantling domains doesnt really justify the NURI change. The path syntax, of `/ipfs/{hash}`, contains functionally the same information.
2) If you still need IPNS, then you still need a concept of domains, so the "there is no domain" premise isn't really accurate.
There's the concept of nestability in NURIs that is supposed to increase protocol composition, but I think you're overgeneralizing, at the cost of breaking backwards compatibility. When we looked at using IPFS in Beaker, the NURI was a major problem for us, because we're limited to what Electron/Chrome provides. It's not just API surface either: there's a lot of code in Chromium that makes assumptions around the standard URL syntax. Are you really so sure that nestable references are worth the headache? Because you're gambling the entire IPFS project on it.
Re: Dat – Distributed Dataset Synchronization and Versioning
#15We use Dat in Beaker[1] to host sites and files from the user's device. It's a pretty interesting protocol. It's developed by Code for Science[2], a 501(c)(3) led by Max Ogden[3] and with protocol dev led by Mafintosh[4]; their mission is to help with archival of science and civic data. Some interesting properties: 1. It uses a BitTorrent-style of swarm, but primarily to sync signed append-only logs, which are in fac…
I wish Beaker had picked a different name. It collides with the Beaker Notebook, a Jupyter alternative that unfortunately never seemed to gain traction but had some really killer features that Jupyter has yet to pick up, especially the ability to mix Python, R, and Julia cells in the same notebook.
Re: Dat – Distributed Dataset Synchronization and Versioning
#16I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been getting the data -- building it requires about 100 GB of raw data or 15 GB of computed data that can be imported into PostgreSQL.
I once tried git-annex. It turned out not to be a good choice -- its tools were flaky, its usage patterns confusing, it leaves a permanent record of your mistakes in configuring data sources, and it was very hard to convince to use ordinary HTTP downloads instead of trying to get read-write access to S3 (which wouldn't work for anyone but me). Now I have weird branches and remotes in my repositories, and weird data in my S3 buckets, that I can't get rid of in case someone tries to use git-annex in a way I told them would work.
After that I just went with distributing the data with plain HTTP downloads from S3. I wish I could do better than this. The only semblance of versioning is putting the date in the URL, and also people in Asia tell me that the build fails because their downloads from us-east-1 get interrupted. Oh, and if I ever stop paying for S3, everything will break.
If I tried making data reproducible with Dat, would it be safe to promise people that they could use Dat to get the data? Even if in the future I don't like Dat anymore?
For instance, do I have to commit to hosting the data somewhere? If not, who does? Does it disappear when people lose interest, like BitTorrent?
Re: Dat – Distributed Dataset Synchronization and Versioning
#17- You can publish new versions to a URL until you somehow forget the private key, and then it's fixed forever, so long as people hang onto copies.
- There's nothing to prevent people from passing around a URL with a version in it. So, although it looks like the author has some control, this is an illusion; publishing is irrevocable and anything published could go viral. (This is generally true of making copies, but it's the opposite of Snapchat.)
- Suppose someone chooses to publish a private key? Is it a world-writable URL? Hmm.
Re: Dat – Distributed Dataset Synchronization and Versioning
#18As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…
Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon.
Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and I can help you get started.
Re: Dat – Distributed Dataset Synchronization and Versioning
#19It seems like Dat has some usability quirks that might take some getting used to: - You can publish new versions to a URL until you somehow forget the private key, and then it's fixed forever, so long as people hang onto copies. - There's nothing to prevent people from passing around a URL with a version in it. So, although it looks like the author has some control, this is an illusion; publishing is irrevocable and…
At the moment that would result in each leaked-key user maintaining a different history, with different peers only downloading the updates from the leak-author they happen to receive data from first. But in the future what will happen, once we get to writing the software for it, is the corruption event will be detected and recorded by all possible peers, freezing the dat from receiving future updates.
Re: Dat – Distributed Dataset Synchronization and Versioning
#20As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…
Just today I was using the multilingual Conceptnet-numberbatch word vectors[1], which would not be possible without your work.
To your point though - you can use Amazon S3 as a seed for Bitorrent downloads, which might help some and reduce what you pay. See [2]
[1] https://github.com/commonsense/conceptnet-numberbatch
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.htm...