Dat – Distributed Dataset Synchronization and Versioning • Hacker News Board

Re: Dat – Distributed Dataset Synchronization and Versioning

#21

Post by hobofan » Fri, May 26, 2017, 7:00 AM UTC

pfraze wrote: post #14

lgierth wrote: post #13

Earlier quoted context omitted.

Actually there's been some movement regarding URLs :) You can check out the reasoning and plan here: [1] IPFS still has the long-term goal of path addressing (NURI), we just hadn't yet completely figured out what the upgrade path should look like. The discussion linked above is turning into a spec and into actions, i.e. IPFS will be trying to get as much as possible of Electron's protocol API [2] into WebExtensions,…

So from what I can tell, the reason y'all keep pushing for NURIs has to do with the focus on hash-addressing, right? You're trying to get away from the concept of domains, which does make sense: a content-addressed folder or blob can live independently of a domain, because it's immutable and self-verifying, so why have domains at all. Here's why I think you're shooting yourself in the foot with the NURIs, though. 1)…

> Because you're gambling the entire IPFS project on it.

One of the reasons I like the IPFS project so much is that they produce a lot of good side products (multiformats, libp2p) that can help anyone build an IPFS-like system.

So I wouldn't see it that bleak. Even if that choice of NURIs is a fatal flaw as you claim, it is still a very surface-level problem that could be fixed in a fork of it.

Re: Dat – Distributed Dataset Synchronization and Versioning

#22

Post by thrw1001 » Fri, May 26, 2017, 9:00 AM UTC

rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

What underlying ontology does it use?

Re: Dat – Distributed Dataset Synchronization and Versioning

#23

Post by vertex-four » Fri, May 26, 2017, 9:30 AM UTC

GhotiFish wrote: post #9

tbv wrote: post #8

Earlier quoted context omitted.

What do you think it sacrifices? IPFS is more focused on static blob addressing while Dat focuses on data sources, and in that sense, there’s a single authority over a dataset. But I see that as a positive, since mutability is pretty valuable. TBH, I'm not sure how a site can work without mutability.

a dynamic site can't work without mutability, ipfs can't deal with mutability, so ipfs comes with ipns, which allows you to statically reference content that might change.

Of course, the design of IPNS makes it impossible to prove that you've got the latest version of a name's value, and makes it relatively easy to attack. I don't know if Dat has the same issue, I haven't looked at it.

Re: Dat – Distributed Dataset Synchronization and Versioning

#24

Post by rspeer » Fri, May 26, 2017, 10:16 AM UTC

nl wrote: post #20

rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

Thankyou for your work on ConceptNet. It's the best public knowledge graph in existence. Just today I was using the multilingual Conceptnet-numberbatch word vectors[1], which would not be possible without your work. To your point though - you can use Amazon S3 as a seed for Bitorrent downloads, which might help some and reduce what you pay. See [2] [1] https://github.com/commonsense/conceptnet-numberbatch [2] http://…

Yep, ConceptNet Numberbatch is my work too, and it's been the most effective way to show that knowledge graphs matter -- that there is more to know about word relationships than you can get from distributional semantics ("word2vec") alone.

Re: Dat – Distributed Dataset Synchronization and Versioning

#25

Post by rspeer » Fri, May 26, 2017, 11:00 AM UTC

pfraze wrote: post #18

rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

Conceptnet looks really cool, I'm going to dig into that later tonight. Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon. Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and…

Should I just try using the tools at https://datproject.org/ ?

It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-sharing the files to myself.)

And I guess a Dat URL shouldn't really be an authoritative entry point referred to in a paper, as it would just point to one version of the data with no context. Maybe Dat plus Zenodo could do the trick eventually.

Re: Dat – Distributed Dataset Synchronization and Versioning

#26

Post by nl » Fri, May 26, 2017, 11:13 AM UTC

rspeer wrote: post #24

nl wrote: post #20

Earlier quoted context omitted.

Thankyou for your work on ConceptNet. It's the best public knowledge graph in existence. Just today I was using the multilingual Conceptnet-numberbatch word vectors[1], which would not be possible without your work. To your point though - you can use Amazon S3 as a seed for Bitorrent downloads, which might help some and reduce what you pay. See [2] [1] https://github.com/commonsense/conceptnet-numberbatch [2] http://…

Yep, ConceptNet Numberbatch is my work too, and it's been the most effective way to show that knowledge graphs matter -- that there is more to know about word relationships than you can get from distributional semantics ("word2vec") alone.

Oh really? Very nice... although I'm only using the aligned distributional semantic nature of them.

I have some background in question answering over knowledge graphs, though, so I'm familiar with their strengths.

Re: Dat – Distributed Dataset Synchronization and Versioning

#27

Post by alexvoda » Fri, May 26, 2017, 12:16 PM UTC

pfraze wrote: post #4

nerdponx wrote: post #3

Earlier quoted context omitted.

I wish Beaker had picked a different name. It collides with the Beaker Notebook, a Jupyter alternative that unfortunately never seemed to gain traction but had some really killer features that Jupyter has yet to pick up, especially the ability to mix Python, R, and Julia cells in the same notebook.

Yeah I regret that. We'll consider a rename at some point. Here's their project: http://beakernotebook.com/

Will you keep the laboratory glassware theme?

Flask is already used ( http://flask.pocoo.org/ )

Maybe Retort? Funnel? Or maybe a proper name? Berzelius? Erlenmeyer would be difficult for people to pronounce.

Re: Dat – Distributed Dataset Synchronization and Versioning

#28

Post by lgierth » Fri, May 26, 2017, 1:29 PM UTC

vertex-four wrote: post #23

GhotiFish wrote: post #9

Earlier quoted context omitted.

a dynamic site can't work without mutability, ipfs can't deal with mutability, so ipfs comes with ipns, which allows you to statically reference content that might change.

Of course, the design of IPNS makes it impossible to prove that you've got the latest version of a name's value, and makes it relatively easy to attack. I don't know if Dat has the same issue, I haven't looked at it.

It's not impossible by design, it's simply a feature that hasn't been implemented so far. IPFS is by design pluggable on all layers and thus theoretically capable of a ton of stuff.

Re: Dat – Distributed Dataset Synchronization and Versioning

#29

Post by justinclift » Fri, May 26, 2017, 1:55 PM UTC

rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

Hmmm, is the data set something which would fairly naturally fit in a series of SQLite databases?

100GB is way too large for the project I'm working on at the moment (dbhub.io), as even a bunch of people downloading something that large would nuke our sponsorship budget since we're just starting out (still pre-launch).

However, if we gain traction and become cash positive, data sets this size would be good to cater to. :)

Re: Dat – Distributed Dataset Synchronization and Versioning

#30

Post by rspeer » Fri, May 26, 2017, 2:39 PM UTC

justinclift wrote: post #29

rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

Hmmm, is the data set something which would fairly naturally fit in a series of SQLite databases? 100GB is way too large for the project I'm working on at the moment (dbhub.io), as even a bunch of people downloading something that large would nuke our sponsorship budget since we're just starting out (still pre-launch). However, if we gain traction and become cash positive, data sets this size would be good to cater t…

I used to keep it in SQLite (much easier to distribute than PostgreSQL). It worked a lot better than many other options I tried. However, rebuilding the database from updated data would take more than a day, and some queries were too slow.

Switching to PostgreSQL sped things up, at the cost of requiring a separate database process, dealing with psql's weird access control, and adding an inconvenient step of loading the data using COPY commands.