Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Version Control

dvc.org

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Data Version Control

#11

Post by matrss »

riedel wrote: post #3
jerednel wrote: post #2

It's not super clear to me how this interacts with data. If I have am using ADLS to store delta tables, and I cannot pull prod to my local can I still use this? Is there a point if I can just look at delta log to switch between past versions?

DVC is (at least as I use it) pretty much just git LFS with multiple backends (guess actually a more simple git annex). It further has some rather MLOps specific stuff. Is handy if you do versions model training with changing data on S3.

Speaking of git-annex, there is another project called DataLad (https://www.datalad.org/), which has some overlap with DVC. It uses git-annex under the hood and is domain-agnostic, compared to the ML focus that DVC has.

Re: Data Version Control

#12

Post by ajoseps »

dmpetrov wrote: post #8

hi there! Maintainer and author here. Excited to see DVC on the front page! Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.

if the data files are all just text files, what are the differences between DVC and using plain git?

Re: Data Version Control

#13

Post by foobarbecue »

andrew_lettuce wrote: post #7
shicholas wrote: post #4

What are the benefits of DVC over Apache Iceberg? If anyone used both, I'd be curious about your take. Thanks!

I don't see any real benefits, as it feels like using the tool you already know even though it's not quite right. Iceberg is maybe geared towards slower changing models than this approach?

username checks out

Re: Data Version Control

#14

Post by dmpetrov »

ajoseps wrote: post #12
dmpetrov wrote: post #8

hi there! Maintainer and author here. Excited to see DVC on the front page! Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.

if the data files are all just text files, what are the differences between DVC and using plain git?

In this cases, you need DVC if:

1. File are too large for Git and Git LFS.

2. You prefer using S3/GCS/Azure as a storage.

3. You need to track transformations/piplines on the file - clean up text file, train mode, etc.

Otherwise, vanilla Git may be sufficient.

Re: Data Version Control

#15

Post by miki123211 »

ajoseps wrote: post #12
dmpetrov wrote: post #8

hi there! Maintainer and author here. Excited to see DVC on the front page! Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.

if the data files are all just text files, what are the differences between DVC and using plain git?

DVC does a lot more than git.

It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically.

There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the intermediate models by accident, not noticing that the script to regenerate it doesn't work any more due to the new dataset structure, and then getting reminded a year later when moving to a new computer and trying to regen everything from scratch.

It's a lot like Unix make, but with the ability to keep track of different git branches and the data / intermediates they need, which saves you from needing to regen everything every time you make a new checkout, lets you easily exchange large datasets with teammates etc.

In theory, you could store everything in git, but then every time you made a small change to your scripts that e.g. changed the way some model works and slightly adjusted a score for each of ten million rows, your diff would be 10m LOC, and all versions of that dataset would be stored in your repo, forever, making it unbelievably large.

Re: Data Version Control

#16

Post by riedel »

haensi wrote: post #10
riedel wrote: post #3

Earlier quoted context omitted.

DVC is (at least as I use it) pretty much just git LFS with multiple backends (guess actually a more simple git annex). It further has some rather MLOps specific stuff. Is handy if you do versions model training with changing data on S3.

There’s another thread from October 2022 on that topic. https://news.ycombinator.com/item?id=33047634 What makes DVC especially useful for MLOps? Aren’t MLFlow or W&B solving that in a way that’s open source (the former) or just increases the speed and scale massively ( the latter)? Disclaimer: I work at W&B.

DVC is much more basic (feels more unix style), integrates really well with any simple CI/CD scripting with git versioning without the need to set up any additional servers.

And it is not either or. People actually combine MLFlow and SVC [0]

[0] https://data-ai.theodo.com/blog-technique/dvc-pipeline-runs-...

Re: Data Version Control

#19

Post by azinman2 »

miki123211 wrote: post #15
ajoseps wrote: post #12

Earlier quoted context omitted.

if the data files are all just text files, what are the differences between DVC and using plain git?

DVC does a lot more than git. It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically. There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the i…

So where do the adjusted 10M rows live instead? S3?
Post reply on HN