Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dat – Distributed Dataset Synchronization and Versioning

github.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Dat – Distributed Dataset Synchronization and Versioning

#31

Post by rspeer »

nl wrote: post #26
rspeer wrote: post #24

Earlier quoted context omitted.

Yep, ConceptNet Numberbatch is my work too, and it's been the most effective way to show that knowledge graphs matter -- that there is more to know about word relationships than you can get from distributional semantics ("word2vec") alone.

Oh really? Very nice... although I'm only using the aligned distributional semantic nature of them. I have some background in question answering over knowledge graphs, though, so I'm familiar with their strengths.

I'd be interested to hear about what you're doing with it.

In my company Luminoso's work, it's important in building domain-specific models that can be used for topic detection, search, and classification. Beyond that, I use it for mostly the basic demos -- word similarity, text similarity, analogies, et cetera.

I believe based on its performance there that it should be a pure upgrade to the kind of applications that use word2vec, but I'd like to know what particular applications it's being used in besides my own.

Re: Dat – Distributed Dataset Synchronization and Versioning

#32

Post by filiwickers »

rspeer wrote: post #25
pfraze wrote: post #18

Earlier quoted context omitted.

Conceptnet looks really cool, I'm going to dig into that later tonight. Dat's really similar to BitTorrent when it comes to availability; it doesn't do anything automatically to guarantee it. If you choose to use Dat, you'll need to ensure a peer exists, though public peer services will be available soon. Dat's still young and you'll probably have to endure some hiccups, but if you do want to give it a try, PM me and…

Should I just try using the tools at https://datproject.org/ ? It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-s…

We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. I'd recommend checking back then (though some, such as CLI, are released already on npm).

Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat compared to http urls.

> Maybe Dat plus Zenodo could do the trick eventually.

Yes, exactly!! Dat supports http publishing right now, so you can run `dat sync --http` on a server and it'll live publish files from the source to an http site.

We are working on also supporting http downloads. The idea is you publish to Zenodo, including the SLEEP metadata, dat can then clone the files over http (including content verification) or via the peer network, i.e. `dat clone zenodo.org/record/439922`.

We are super excited for the http downloading because it'll allow dat to store files on any data repository with a good api and a http file backend. We've been talking with the Dataverse folks on how to accomplish this there and have an eye on others such as Zenodo.

Re: Dat – Distributed Dataset Synchronization and Versioning

#33

Post by datamonsteryum »

rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

I see you’re using JSON-LD in conceptnet. If you start passing that data around using distributed systems, you'll inevitably want to start incorporating content-addressed links into the data. I recommend looking into IPLD as a data model for handling that. https://ipld.io The spec is still open -- this would be a good time to give feedback and/or spell out your use cases in this space.

Re: Dat – Distributed Dataset Synchronization and Versioning

#34

Post by vertex-four »

lgierth wrote: post #28
vertex-four wrote: post #23

Earlier quoted context omitted.

Of course, the design of IPNS makes it impossible to prove that you've got the latest version of a name's value, and makes it relatively easy to attack. I don't know if Dat has the same issue, I haven't looked at it.

It's not impossible by design, it's simply a feature that hasn't been implemented so far. IPFS is by design pluggable on all layers and thus theoretically capable of a ton of stuff.

So IPFS is, according to you, not defined as its protocol, but instead as its API? So I could build an IPFS implementation that just grabs stuff from my (centralised) web server and say I'm using IPFS? Either nonsense, or useless - the goal here, I thought, was to build a global decentralised filesystem that looks the same from everyone's perspective.

All DHTs suffer from this issue. It's just particularly likely that IPNS's use of a DHT will lead to attacks. Making it not susceptible to this would require a redesign of IPNS's protocol.

Re: Dat – Distributed Dataset Synchronization and Versioning

#35

Post by maxogden »

skybrian wrote: post #17

It seems like Dat has some usability quirks that might take some getting used to: - You can publish new versions to a URL until you somehow forget the private key, and then it's fixed forever, so long as people hang onto copies. - There's nothing to prevent people from passing around a URL with a version in it. So, although it looks like the author has some control, this is an illusion; publishing is irrevocable and…

Great analysis. We anticipate that in order to fix these three usability issues around trust we will need to provide a centralized identity provider in the future. This would also address privacy issues especially regarding leaking what dats you are accessing. The design philosophy around Dat is to start from the end of the completely decentralized spectrum but be flexible in letting the application choose the tradeoffs as they move more towards centralized components.

Re: Dat – Distributed Dataset Synchronization and Versioning

#36

Post by justinclift »

rspeer wrote: post #30
justinclift wrote: post #29

Earlier quoted context omitted.

Hmmm, is the data set something which would fairly naturally fit in a series of SQLite databases? 100GB is way too large for the project I'm working on at the moment (dbhub.io), as even a bunch of people downloading something that large would nuke our sponsorship budget since we're just starting out (still pre-launch). However, if we gain traction and become cash positive, data sets this size would be good to cater t…

I used to keep it in SQLite (much easier to distribute than PostgreSQL). It worked a lot better than many other options I tried. However, rebuilding the database from updated data would take more than a day, and some queries were too slow. Switching to PostgreSQL sped things up, at the cost of requiring a separate database process, dealing with psql's weird access control, and adding an inconvenient step of loading t…

Hmmm, sounds like the data itself would be feasible then. SQLite could be considered just a data transport container for this purpose. :)

Re: Dat – Distributed Dataset Synchronization and Versioning

#37

Post by skybrian »

maxogden wrote: post #35
skybrian wrote: post #17

It seems like Dat has some usability quirks that might take some getting used to: - You can publish new versions to a URL until you somehow forget the private key, and then it's fixed forever, so long as people hang onto copies. - There's nothing to prevent people from passing around a URL with a version in it. So, although it looks like the author has some control, this is an illusion; publishing is irrevocable and…

Great analysis. We anticipate that in order to fix these three usability issues around trust we will need to provide a centralized identity provider in the future. This would also address privacy issues especially regarding leaking what dats you are accessing. The design philosophy around Dat is to start from the end of the completely decentralized spectrum but be flexible in letting the application choose the tradeo…

Good to know.

It would be good to figure out key rotation for Dat URL providers since this probably has to be built into the protocol.

Any thoughts on integrating with keybase? I like keybase's model where you have device-specific keys. But this would probably make moving a Dat URL provider to a different machine trickier.

This all assumes that well-known Dat URL's become an important thing to preserve (they are published in papers, etc) even though they are very user-unfriendly, even more than IP addresses.

A naming system on top of them would make key rotation a non-issue (rotate Dat URL's instead) and you could completely replace or remove the history, sort like a git rebase. But that loses other nice properties of the system?

I suppose irrevocability is something we deal with in git repos all the time. Although you can do a rebase locally, once a commit is accepted by a popular project, they're unlikely to let you remove it from history. The review process makes it unlikely that any really embarrassing mistake would be accepted, so this seems ok in practice.

Re: Dat – Distributed Dataset Synchronization and Versioning

#38

Post by lgierth »

filiwickers wrote: post #32
rspeer wrote: post #25

Earlier quoted context omitted.

Should I just try using the tools at https://datproject.org/ ? It sounds like Dat is a ways off from being something I could use as an authoritative source of data, but I could include it as one way to get the ConceptNet data. If it succeeds, that could save on the S3 bill and maybe even distribute the data across continents better. (Heck, I'm sure a lot of the downloads are scripts I'm running, and I could be file-s…

We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper. I'd recommend checking back then (though some, such as CLI, are released already on npm). Your points are spot on and things we've been thinking a lot about. I also wouldn't feel comfortable putting a Dat link in a paper yet, but that is an eventual goal because of the persistence properties of dat c…

> We are releasing updated desktop & command line app on Tuesday with support for the protocol, as defined in the paper.

Awesome, congratulations! :):)

For distributed/p2p software these are just amazing times to be alive :)

Re: Dat – Distributed Dataset Synchronization and Versioning

#39

Post by rspeer »

thrw1001 wrote: post #22
rspeer wrote: post #16

As someone who creates open, medium-sized, reusable datasets, is Dat something I should try? Is it too early? The linked page is very much about technical details of the implementation and not about how one would typically use it. I maintain ConceptNet [1], a multilingual knowledge graph. I do everything I can to make its published results reproducible. The biggest hurdle for people reproducing it has always been get…

What underlying ontology does it use?

Its own, I guess? ConceptNet is really not so much about having an upper ontology, it's about relations between natural language words and phrases. Its set of relations is effectively a superset of WordNet's.

Re: Dat – Distributed Dataset Synchronization and Versioning

#40

Post by thrw1001 »

rspeer wrote: post #39
thrw1001 wrote: post #22

Earlier quoted context omitted.

What underlying ontology does it use?

Its own, I guess? ConceptNet is really not so much about having an upper ontology, it's about relations between natural language words and phrases. Its set of relations is effectively a superset of WordNet's.

Hmm but for example. If I search "harry potter", one of the things I get is "harry potter is defined as... " "boy who lives under the stair". What can I do with this? "boy", "lives", "under", and "stair", while present in conceptnet, have no logical relation to anything else...
Post reply on HN