Can anyone compare this to DataLad [1], which someone introduced to me as "git for data"? [ https://www.datalad.org/ ]
And what about Dolt?
Data Version Control
41–50 of 63 posts
Re: Data Version Control
#42Earlier quoted context omitted.
I'm curious, what's the problem with git-annex? I've considered using it before as an alternative to Git LFS.
It lives in this weird wiki that seems to be read-only most of the time. I don't think it's alive. Its use of hard links also causes too many problems, of the silent corruption variety.
Ikiwiki’s definitely a bit weird, but I’ve been experimenting with git-annex recently and it worked fine every time I commented. Seems like it’s chugging along: https://git-annex.branchable.com/recentchanges/
When does it use hard links? As far as I remember it used symlinks unless you used something like annex.hardlink (described in the man page: https://git-annex.branchable.com/git-annex/)
Re: Data Version Control
#43I don't think this tool can encompass everything you need in managing ML models and data sets, even if you limit it to versioning data.
I'd need such a tool to manage features, checkpoints and labels. This doesn't do any of that. Nor does it really handle merging multiple versions of data.
And I'd really like the code to be handled separately from the data. Git is not the place to do this. Because the choice of picking pairs of code and data should happen at a higher level, and be tracked along with the results - that's not going in a repo - MLFlow or Tensorboard handles it better.
Re: Data Version Control
#44Earlier quoted context omitted.
Yes it definitely serves a valid use-case, I feel like someone should try and bring some competition there. A modern equivalent with fewer gotchas, maybe in Rust/Go, maybe using a fuse mount and content-defined chunking (borg/restic/...-style) would be amazing.
I'd love to see a well-supported git-lfs compatible client/proxy (so you could more easily move backends) that could run on top of S3/object storage. Yes, and written in a modern language like golang/rust for performance / parallelism. There's some node.js and various other git-lfs proxies out there, but not well enough maintained that I could count on them being around and working in another 5 years. git-annex at le…
Did some more research to see if anything had changed in this space. I found two interesting projects (haven't used them myself yet though):
One in C# (with support for auth)
https://github.com/alanedwardes/Estranged.Lfs
One in Rust (but no Auth, have to run reverse proxy)
https://github.com/jasonwhite/rudolfs
Both seem interesting. Anyone use these?
Re: Data Version Control
#45Earlier quoted context omitted.
I'm curious, what's the problem with git-annex? I've considered using it before as an alternative to Git LFS.
things that I don't like about it: * git diff doesn't work in any sensible way * if you forget and do `git add` instead of `git annex add`, everything is fine, but you've now spoilt the nice thing that git annex does of de-duping files. (git annex only stores one copy of identical files) * for our use case (which I'm sure is the wrong way of doing things) it's possible to overwrite the single copy of a file that git…
If you configure annex.largefiles, git add should work with the annex. I start with something like
git annex config --set annex.largefiles 'largerthan=1kb and not (mimeencoding=us-ascii or mimeencoding=utf-8)'
Note that git add will add large files unlocked, though, since (as far as I understand) it’s assumed you’re still modifying them for safety:
> If you use git add to add a file to the annex, it will be added in unlocked form from the beginning. This allows workflows where a file starts out unlocked, is modified as necessary, and is locked once it reaches its final version. —https://git-annex.branchable.com/git-annex-unlock/
Re: Data Version Control
#46Can anyone compare this to DataLad [1], which someone introduced to me as "git for data"? [ https://www.datalad.org/ ]
And what about Dolt? https://docs.dolthub.com/introduction/what-is-dolt
Dolt is for tabular data. It's like SQLite but with branching, versioning of the DB level. DVC is file-based. It saves large files, directories, etc to one of the supported storages - S3, GCP, Azure, etc. It's more like Git-lfs in that sense.
Another difference is that for DVC (surprisingly) data versioning itself is just one of the main fundamental layers that is needed to provide holistic ML experiments tracking and versioning. So, DVC has a layer to describe an ML project, run it, capture and version inputs/outputs. In that sense DVC becomes a more opinionated / high level tool if that makes sense.
Re: Data Version Control
#47If you're looking for something that actually tracks tabular data there's https://kartproject.org. Geo focused but also works with standard database tables. Built with git (kart repos are git repos), can track PostgreSQL, MSSQL, MySQL etc.
Re: Data Version Control
#48If you just want a git for large data files, and your files don't get updated too often (e.g. an ML model deployed in production which gets updated every month) then git-lfs is a nice solution. Bitbucket and Github both have support for it.
+1. git-lfs is sufficient for tracking binaries, including a ML model, at that cadence.
Thinking more abstractly, there is benefit for code and data to live "next" to each other, if possible. Atomically committed to a codebase and the latter loaded / used by the former without connecting to yet another workflow.
Re: Data Version Control
#49Earlier quoted context omitted.
> But the truth is that DVC is not a great tool for running "experiments" searching over a parameter space. Would love your feedback what's missing there! We've been improving it lately - e.g. - Hydra support https://dvc.org/doc/user-guide/experiment-management/hydra - VS Code extension - https://marketplace.visualstudio.com/items?itemName=Iterativ...
Last I checked it wasn't easy to use something like optuna to do hyperparameter tuning with hydra/DVC. Ideally I'd like the tool I use for data versioning (DVC/git-lfs/gif-annex) to be orthogonal to that which I use for hyperparameter sweeping (DVD/optuna/SageMaker experiments), and orthogonal to that which I use for configuration management (DVC/Hydra/Plain YAML), to that what I use for experimental DAG management (…
Big +1 about composability and orthogonality. I don't want one "do it all" tool, I want a collection of small tools that interoperate nicely. Like how you can use Airflow and DBT together, but neither tool really tries to do what the other one does (not that Airflow is "small", but still).
Re: Data Version Control
#50Earlier quoted context omitted.
It lives in this weird wiki that seems to be read-only most of the time. I don't think it's alive. Its use of hard links also causes too many problems, of the silent corruption variety.
Ikiwiki’s definitely a bit weird, but I’ve been experimenting with git-annex recently and it worked fine every time I commented. Seems like it’s chugging along: https://git-annex.branchable.com/recentchanges/ When does it use hard links? As far as I remember it used symlinks unless you used something like annex.hardlink (described in the man page: https://git-annex.branchable.com/git-annex/ )
Symlinks are just as problematic honestly, an app writing to it will change the object in the persistent "immutable" storage. The way the "check out" feature works is also weird, causing a change in the shared version history.