Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dat – A git-like tool for large datasets

dat-data.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Dat – A git-like tool for large datasets

#31

Post by TillE »

derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

You could probably modify Mercurial's largefiles extension to use S3, etc as a central store.

Re: Dat – A git-like tool for large datasets

#32

Post by qwerta »

Camillo wrote: post #25

I am surprised at how far along the PR effort is compared to the actual product development. The code seems to be still in its embryonic stage, and most of the features are simply desiderata at this point; yet there is already a catchy name, a snazzy logo, a nice website, and even stickers! And this Hacker News post, of course. Now, it is possible that this happened by chance: for instance, perhaps the author has a f…

I have opposite problem. My project is almost production ready, but I have not even published performance benchmarks yet :-)

I would find helpful if author would outline how PR is done.

Re: Dat – A git-like tool for large datasets

#33

Post by beagle3 »

derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

Libgit2 does this internally, although it doesn't expose the underlying store (so you wouldn't know to use S3 even if you could get objects from there)

Re: Dat – A git-like tool for large datasets

#34

Post by t1m »

Git is pretty good at handling large files right now.

We use git to push configuration data to (near) real-time systems that keep this data in local memory mapped key-value stores. It's essentially a fully replicated, eventually consistent key-value store, and git plays a staring role.

We regularly push multi-gigabyte files through this system with ease, with only a few tweaks on git's configuration. Git has some major advantages. It's fast, can handle large files, and you can piggyback on the versioning system to ensure that multiple writers aren't competing. Also, git is highly configurable and has a lot of out-of-the-box options for easily setting up git servers.

Re: Dat – A git-like tool for large datasets

#35

Post by handelaar »

t1m wrote: post #34

Git is pretty good at handling large files right now. We use git to push configuration data to (near) real-time systems that keep this data in local memory mapped key-value stores. It's essentially a fully replicated, eventually consistent key-value store, and git plays a staring role. We regularly push multi-gigabyte files through this system with ease, with only a few tweaks on git's configuration. Git has some maj…

Git, in common with every other version control system I ever used, considers adding a column to a CSV file to be "every line was altered" because Lines are all-important.

Re: Dat – A git-like tool for large datasets

#36

Post by mtrn »

_prometheus wrote: post #24
kcorbitt wrote: post #23

Dat is really exciting. A common format for sharing and hosting datasets, to say nothing of its versioning and transforming of data, is really necessary right now. I've been working on an open-source site that will act kind of like a Github for Dat[0][1]. I started four months ago, but decided that dat was still too unstable at the time to really build an API client against, so I'm waiting for at least an alpha relea…

Hey kcorbitt, cool stuff. I'm working with Max on dat. Before I knew about dat, I built http://datadex.io (which is like an npm for datasets -- example: http://datadex.io/jbenet/cifar-100 ). datadex will support dat data first class. Would be good to get your help when all this happens. :)

I like the Datadex idea and the promise to make datasets as easily accessible as source code repositories. I can see a number of use cases already for a standardized and fast way to collaborate on structured data sets.

Re: Dat – A git-like tool for large datasets

#37

Post by drewda »

Camillo wrote: post #25

I am surprised at how far along the PR effort is compared to the actual product development. The code seems to be still in its embryonic stage, and most of the features are simply desiderata at this point; yet there is already a catchy name, a snazzy logo, a nice website, and even stickers! And this Hacker News post, of course. Now, it is possible that this happened by chance: for instance, perhaps the author has a f…

DAT is funded by the Knight Foundation: http://www.knightfoundation.org/grants/201346305/

Re: Dat – A git-like tool for large datasets

#38

Post by prayag »

Camillo wrote: post #25

I am surprised at how far along the PR effort is compared to the actual product development. The code seems to be still in its embryonic stage, and most of the features are simply desiderata at this point; yet there is already a catchy name, a snazzy logo, a nice website, and even stickers! And this Hacker News post, of course. Now, it is possible that this happened by chance: for instance, perhaps the author has a f…

I am sorry but are we looking at the same website. I see two static pages and a logo. Compared to more than 2 months of work that has been put in the code base by what seems like a very talented team, the efforts they have made on the "PR" is minimal (a few hours, probably even less). I think you are being extremely unfair.

Re: Dat – A git-like tool for large datasets

#39

Post by skybrian »

t1m wrote: post #34

Git is pretty good at handling large files right now. We use git to push configuration data to (near) real-time systems that keep this data in local memory mapped key-value stores. It's essentially a fully replicated, eventually consistent key-value store, and git plays a staring role. We regularly push multi-gigabyte files through this system with ease, with only a few tweaks on git's configuration. Git has some maj…

How do you deal with old data, like multigigabye files deleted last year that everyone has to download because it's git?

Re: Dat – A git-like tool for large datasets

#40

Post by sciurus »

beliu wrote: post #26
derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

See also: https://camlistore.org/ (unlike Dat, doesn't target datasets, but more generic user data, like photos and files; more Dropbox than npm). It has a pretty cool blob model, exposes a FUSE interface, has a slick photo-browsing UI, and integrates with S3 and (I think) Glacier.

For a recent look at camlistore see http://lwn.net/SubscriberLink/595746/2adfdecdb0de2577/

(and if you like that article, consider subscribing to LWN!)

Post reply on HN