Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dat – A git-like tool for large datasets

dat-data.com

Post reply on HN
Print view View original

11–20 of 43 posts

Re: Dat – A git-like tool for large datasets

#11

Post by rpedela »

derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

Isn't that what git-annex is for?

https://git-annex.branchable.com/

Re: Dat – A git-like tool for large datasets

#12

Post by michaelmior »

derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

You would end up with pretty huge latency for a lot of common operations in that case. Every checkout, diff, and status for example.

Re: Dat – A git-like tool for large datasets

#13

Post by zobzu »

michaelmior wrote: post #12
derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

You would end up with pretty huge latency for a lot of common operations in that case. Every checkout, diff, and status for example.

yeah it sounds like svn all over again

Re: Dat – A git-like tool for large datasets

#15

Post by maxogden »

Hi, maintainer of dat here. We're working on the first release now. The first use case is tabular data (a large volume of small rows), but we'll be adding a content-addressable blob store (for big binary attachments) on top for the beta, as well as some fun p2p (webrtc data channel) powered sync stuff (will be somewhat similar to BitTorrent sync) to maximize download speeds and let universities seed big scientific datasets.

Re: Dat – A git-like tool for large datasets

#16

Post by maxogden »

sheetjs wrote: post #14

Is there any sort of Excel integration? The usage documents reference dat-excel, but I don't see any code

not yet, but keep an eye on https://github.com/jbenet/transformer which will provide the data format conversion API. It would be really cool to see some sheetjs stuff hooked up to dat/transformer!

Re: Dat – A git-like tool for large datasets

#17

Post by _prometheus »

maxogden wrote: post #15

Hi, maintainer of dat here. We're working on the first release now. The first use case is tabular data (a large volume of small rows), but we'll be adding a content-addressable blob store (for big binary attachments) on top for the beta, as well as some fun p2p (webrtc data channel) powered sync stuff (will be somewhat similar to BitTorrent sync) to maximize download speeds and let universities seed big scientific da…

if people are interested in the p2p blob store, poke me to write it faster. github.com/jbenet

Re: Dat – A git-like tool for large datasets

#18

Post by espeed »

derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

You can use JGit (https://github.com/eclipse/jgit) to store to S3.

See http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...

Re: Dat – A git-like tool for large datasets

#19

Post by derefr »

michaelmior wrote: post #12
derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

You would end up with pretty huge latency for a lot of common operations in that case. Every checkout, diff, and status for example.

That depends on how replicated/distributed the object storage is. In a DHT scenario, you might already be storing some of the objects yourself before you've even requested them.

The remote server can obviously be running an object-storage node itself, and so can your own local git repo -- the point is that you can have objects in more places than just your machine and the repo it's pulling from.

Re: Dat – A git-like tool for large datasets

#20

Post by platz »

rpedela wrote: post #11
derefr wrote: post #10

I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem,…

Isn't that what git-annex is for? https://git-annex.branchable.com/

Also some interesting related projects on https://git-annex.branchable.com/not/
Post reply on HN