Dat – A git-like tool for large datasets
dat-data.com
Dat – A git-like tool for large datasets
1–10 of 43 posts
Re: Dat – A git-like tool for large datasets
#2I can't find any information on the tools capabilities or implementation on that website. The README on github isn't helpful either.
Re: Dat – A git-like tool for large datasets
#3I can't find any information on the tools capabilities or implementation on that website. The README on github isn't helpful either.
There is another readme that is more informative:
Re: Dat – A git-like tool for large datasets
#4I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores.
I could be missing something here so anybody more knowledgeable should chime in.
Re: Dat – A git-like tool for large datasets
#5I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores. I could be missing something here so anybody more knowledgeable should chime in.
It might not be optimal; however I wonder if most of the primary sources this is intended to target (i.e. govt) prefers to use textual data?
Re: Dat – A git-like tool for large datasets
#6I can't find any information on the tools capabilities or implementation on that website. The README on github isn't helpful either.
Yeah, they're hidden away in their git repo [1]. They have a good set of example usage commands [2], and a nice introductory document explaining what dat is [3]. Perhaps most important are the technical notes [4] that describe the supported formats.
[1]: https://github.com/maxogden/dat
[2]: https://github.com/maxogden/dat/blob/master/usage.md
[3]: https://github.com/maxogden/dat/blob/master/what-is-dat.md
Re: Dat – A git-like tool for large datasets
#7I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores. I could be missing something here so anybody more knowledgeable should chime in.
It might not be optimal; however I wonder if most of the primary sources this is intended to target (i.e. govt) prefers to use textual data?
>I wonder if most of the primary sources this is intended to target (i.e. govt) ...
The second sentence on their website appears to state otherwise: "As a team we have a bias towards supporting scientific + research data use cases."
Edit: Well, on further reading they do go on to say, "What we're building isn't quite ready for prime time yet, but if you want to play around...", so perhaps I should not jump the gun :)
Re: Dat – A git-like tool for large datasets
#8I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores. I could be missing something here so anybody more knowledgeable should chime in.
[deleted]
Re: Dat – A git-like tool for large datasets
#9I don't think this is the right approach. I'm fairly certain, after reading their website and git repo, that dat really does only deal with textual data. As someone that deals with huge datasets on a daily basis, text files are really out of the question. I have to rely on netCDF or HDF5 for my data stores. I could be missing something here so anybody more knowledgeable should chime in.
It might not be optimal; however I wonder if most of the primary sources this is intended to target (i.e. govt) prefers to use textual data?
Nope. Any really large government data is not text -- it's mostly in older open binary formats (e.g. weather data in GRIB from NOAA) or expensive proprietary binary formats (e.g. healthcare data in SAS from CMS).
Re: Dat – A git-like tool for large datasets
#10I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem, especially if you also allowed composite file objects, and perhaps per-object concurrent BitTorrent retrieval.