Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Version Control

dvc.org

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Data Version Control

#31

Post by shcheklein »

bs7280 wrote: post #24

What value does this provide that I can't get by versioning my data in partitioned parquet files on s3?

I think parquet won't help with images, video, ML models.

Also, one thing is to physically provide a way to version data (e.g. partitioned parquet files, cloud versioning, etc, etc), but another one is to also have a mechanism of saving / codifying dataset version into the project. E.g. to answer the question which version of data this model was built with you would need to save some identifier / hash / list of files that were used. DVC takes care of that part as well.

(it has mechanics to cache data that you download, make-file like pipelines, etc)

Re: Data Version Control

#32

Post by remram »

imiric wrote: post #21
remram wrote: post #11

Earlier quoted context omitted.

Doesn't use git-annex like DataLad. That alone is a huge benefit given the state of that tool.

I'm curious, what's the problem with git-annex? I've considered using it before as an alternative to Git LFS.

It lives in this weird wiki that seems to be read-only most of the time. I don't think it's alive. Its use of hard links also causes too many problems, of the silent corruption variety.

Re: Data Version Control

#33

Post by remram »

niccl wrote: post #30
imiric wrote: post #21

Earlier quoted context omitted.

I'm curious, what's the problem with git-annex? I've considered using it before as an alternative to Git LFS.

things that I don't like about it: * git diff doesn't work in any sensible way * if you forget and do `git add` instead of `git annex add`, everything is fine, but you've now spoilt the nice thing that git annex does of de-duping files. (git annex only stores one copy of identical files) * for our use case (which I'm sure is the wrong way of doing things) it's possible to overwrite the single copy of a file that git…

Yes it definitely serves a valid use-case, I feel like someone should try and bring some competition there. A modern equivalent with fewer gotchas, maybe in Rust/Go, maybe using a fuse mount and content-defined chunking (borg/restic/...-style) would be amazing.

Re: Data Version Control

#34

Post by simonw »

adhocmobility wrote: post #19

If you just want a git for large data files, and your files don't get updated too often (e.g. an ML model deployed in production which gets updated every month) then git-lfs is a nice solution. Bitbucket and Github both have support for it.

It seems to be the solution Hugging Face have picked too.

Re: Data Version Control

#35

Post by shcheklein »

nerdponx wrote: post #17
remram wrote: post #10

Earlier quoted context omitted.

> You are encouraged if you model your pipeline in DVC. Encouraged to do what? You might want to slow down on the use of parentheses, we are both getting lost in them.

I assume they meant to say "you are encouraged to use DVC to run your model and experiment pipeline". They want to encourage you to do this because they are trying to build a business around being a data science ops ecosystem. But the truth is that DVC is not a great tool for running "experiments" searching over a parameter space. it could be improved in that regard, but that's just not what I use it for nor is it wh…

> But the truth is that DVC is not a great tool for running "experiments" searching over a parameter space.

Would love your feedback what's missing there! We've been improving it lately - e.g.

- Hydra support https://dvc.org/doc/user-guide/experiment-management/hydra

- VS Code extension - https://marketplace.visualstudio.com/items?itemName=Iterativ...

Re: Data Version Control

#36

Post by kernelsanderz »

adhocmobility wrote: post #19

If you just want a git for large data files, and your files don't get updated too often (e.g. an ML model deployed in production which gets updated every month) then git-lfs is a nice solution. Bitbucket and Github both have support for it.

I do feel like git-lfs is a good solution. Once you have 10s or 100s of GB of files (eg. a computer vision project), this gets pretty pricey.

Ideally I'd love to use git-lfs on top of S3, directly. I've looked into git-annex and various git-lfs proxies, but I'm not sure they're maintained well enough to be trusting it with long-term data storage.

Huggingface datasets are built on git-lfs and it works really well for them for storage of large datasets. Ideally I'd love for AWS to offer this as a hosted thin layer on top of S3, or for some well funded or supported community effort to do the same, and in a performant way.

If you know of any such solution, please let me know!

Re: Data Version Control

#37

Post by kernelsanderz »

remram wrote: post #33
niccl wrote: post #30

Earlier quoted context omitted.

things that I don't like about it: * git diff doesn't work in any sensible way * if you forget and do `git add` instead of `git annex add`, everything is fine, but you've now spoilt the nice thing that git annex does of de-duping files. (git annex only stores one copy of identical files) * for our use case (which I'm sure is the wrong way of doing things) it's possible to overwrite the single copy of a file that git…

Yes it definitely serves a valid use-case, I feel like someone should try and bring some competition there. A modern equivalent with fewer gotchas, maybe in Rust/Go, maybe using a fuse mount and content-defined chunking (borg/restic/...-style) would be amazing.

I'd love to see a well-supported git-lfs compatible client/proxy (so you could more easily move backends) that could run on top of S3/object storage. Yes, and written in a modern language like golang/rust for performance / parallelism. There's some node.js and various other git-lfs proxies out there, but not well enough maintained that I could count on them being around and working in another 5 years. git-annex at least has been around for a while, even though it has its issues.

Huggingface uses git-lfs for large datasets with good success. git-lfs on GitHub gets very pricey at higher volumes of data. Would love the affordability of object storage, just with a better git blob storage interface, that will be around in the future.

Most of these systems do their own hash calculations and are not interchangeable with each other. I feel like git-lfs has the momentum at the momentum in data-science at the moment, but needs some better options for people who want a low cost storage option that they can control.

Huggingface is great, but it's one more service to onboard if you're in an enterprise. And data privacy/retention/governance means that many people would liek their data to reside on their own infrastructure.

If AWS were to give us a low cost git-lfs hosted service on top of S3 it would be very popular.

If anyone knows of some good alternatives, please let us know!

Re: Data Version Control

#38

Post by kernelsanderz »

shcheklein wrote: post #35
nerdponx wrote: post #17

Earlier quoted context omitted.

I assume they meant to say "you are encouraged to use DVC to run your model and experiment pipeline". They want to encourage you to do this because they are trying to build a business around being a data science ops ecosystem. But the truth is that DVC is not a great tool for running "experiments" searching over a parameter space. it could be improved in that regard, but that's just not what I use it for nor is it wh…

> But the truth is that DVC is not a great tool for running "experiments" searching over a parameter space. Would love your feedback what's missing there! We've been improving it lately - e.g. - Hydra support https://dvc.org/doc/user-guide/experiment-management/hydra - VS Code extension - https://marketplace.visualstudio.com/items?itemName=Iterativ...

Last I checked it wasn't easy to use something like optuna to do hyperparameter tuning with hydra/DVC.

Ideally I'd like the tool I use for data versioning (DVC/git-lfs/gif-annex) to be orthogonal to that which I use for hyperparameter sweeping (DVD/optuna/SageMaker experiments), and orthogonal to that which I use for configuration management (DVC/Hydra/Plain YAML), to that what I use for experimental DAG management (DVC/Makefile)

Optuna is becoming very popular in the data-science/deep learning ecosystem at the moment. It would be great to see more composable tools, rather than having to opt all-in into a given ecosystem.

Love the work that DVC is doing though to tackle these difficult problems though!

Re: Data Version Control

#39

Post by kortex »

adhocmobility wrote: post #19

If you just want a git for large data files, and your files don't get updated too often (e.g. an ML model deployed in production which gets updated every month) then git-lfs is a nice solution. Bitbucket and Github both have support for it.

I've used both extensively. Git-lfs has always been a nightmare. Because each tracked large file can be in one of two states - binary, or "pointer" - it's super easy for the folder to get all fouled up. It would be unable to "clean" or "smudge", since either would cause some conflict. If you accidentally pushed in the wrong state, you could "infect" the remote and be really hosed. I had this happen numerous times over about 2 years of using lfs, and each time the only solution was some aggressive rewriting of history.

That, combined with the nature of re-using the same filename for the metadata files, meant that it was common for folks to commit the binary and push it. Again, lots of history rewriting to get git sizes back down.

Maybe there exist solutions to my problems but I had spent hours wrestling with it trying to fix these bad states, and it caused me much distress.

Also configuring the backing store was generally more painful, especially if you needed >2GB.

DVC was easy to use from the first moment. The separate meta files meant that it can't get into mixed clean/smudge states. If you aren't in a cloud workflow already, the backing store was a bit tricky, but even without AWS I made it work.

Re: Data Version Control

#40

Post by kortex »

AlotOfReading wrote: post #20
nerdponx wrote: post #16

Earlier quoted context omitted.

DVC is expressly for tracking artifacts that are files on disk, and only by comparing their MD5 hashes. So it can definitely track the parquet files, but you are not going to get row or field diffs or anything like that. Maybe Pachyderm or Dolt would be better tools here.

Why would you use MD5 in anything written in the last 5 years? The SHA family is faster on modern hardware and there aren't trivial collisions floating around out there.

It was definitely a bad choice. I wasn't there so I can only speculate. My guess is because it is sort of ubiquitous and thus a low-hanging fruit and devs didn't know better, or the related corollary, it's what S3 uses for ETags, so it probably seemed logical. Either way, seems like someone did it and didn't know better, no one agrees on a fix or whether it's even necessary to change, and thus it's stuck for now.

There's an ongoing discussion about replacing/configuring the hash function, and it looks like there might be some movement toward replacing the hash and other speedups in 3.0

https://github.com/iterative/dvc/issues/3069

> We not only want to switch to a different algorithm in 3.0, but to also provide better performance/ui/architecture/ecosystem for data management, and all of that while not seizing releases with new features (experiements, dvc machine, plots, etc) and bug fixes for 2.0, so we've been gradually rebuilding that and will likely be ready for 3.0 in the upcoming months. - https://github.com/iterative/dvc/issues/3069#issuecomment-93...

Post reply on HN