Dat – A git-like tool for large datasets

git-annex is an ugly hack to, basically, reimplement git-with-this-feature on top of git. What I'm suggesting would be to reimplement git itself on top of distributed object storage -- effectively, to split git into two tools:

1. one that commits treeish diffs into objects, and then stuffs them into arbitrary content-addressable storage;

2. and one that manipulates (local or remote) mutable tables of refs to content-hash-URNs -- or "git repositories" for short.

The key component would be a facility (either a library or an OS service) for resolving content-hash-URNs into file-descriptors in some content-addressable storage system. The critical point is that this would not be a git component -- the committing tool wouldn't be in charge of where its objects go, and the repository tool wouldn't be in charge of resolving their URNs. Instead, both would just be relying on the host to decide how to get objects from/put objects into storage, and it's the host that would have a distribution strategy set up for all of its content-addressable objects, not just the ones that happen to be attached to git.

Dat is really exciting. A common format for sharing and hosting datasets, to say nothing of its versioning and transforming of data, is really necessary right now. I've been working on an open-source site that will act kind of like a Github for Dat[0][1]. I started four months ago, but decided that dat was still too unstable at the time to really build an API client against, so I'm waiting for at least an alpha release before going further. Super excited about the project though.

Currently, the site is just a bare-bones version that lets you submit the URL of a publicly available dataset, rather than including any actual dat integration. I'm planning on changing that though once dat becomes just a bit more stable.

[0] source: https://github.com/kcorbitt/datrepo [1] current site: https://datrepo.com/about

I am surprised at how far along the PR effort is compared to the actual product development. The code seems to be still in its embryonic stage, and most of the features are simply desiderata at this point; yet there is already a catchy name, a snazzy logo, a nice website, and even stickers! And this Hacker News post, of course.

Now, it is possible that this happened by chance: for instance, perhaps the author has a friend who is a graphic designer, and this friend got really excited about the project and put together all of this as a favor. But perhaps it is a deliberate strategy, in which case I wonder: what are the benefits of front-loading the PR work like this? Did it help in securing grants, for instance?

Substack (yes, that substack) discussed this recently:

> a slick web page - this is very often (but not always) a sign of a library that put all of its time into slick marketing but has overly-broad scope and is very likely to become abandoned

http://www.reddit.com/r/javascript/comments/2378xo/the_best_...

just plain git with also large datafiles triggers my mind a lot. not databases ... just all kinds of data. (like large pdfs) ..