Can anyone compare this to DataLad [1], which someone introduced to me as "git for data"? [ https://www.datalad.org/ ]
Doesn't use git-annex like DataLad. That alone is a huge benefit given the state of that tool.
Data Version Control
11–20 of 63 posts
Re: Data Version Control
#12DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it…
DVC is great for use cases that don't get to this scale or have these needs. And the issues here are non-trivial to solve. I've spent a lot of time figuring out how to solve them in Pachyderm which is good for use cases where you do need higher levels of scale or might run into merge conflicts with DVC. There's trade-offs though. DVC is definitely easier for a single developer / data scientist to get up and running with.
Re: Data Version Control
#13DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it…
The alternative tool you are referring to is `Dud` I believe
Dvc is the best tool (I found) inspite of being dead slow and complex (trying to do many things).
What alternatives would you recommend?
Re: Data Version Control
#14DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it…
DVC is great for use cases that don't get to this scale or have these needs. And the issues here are non-trivial to solve. I've spent a lot of time figuring out how to solve them in Pachyderm which is good for use cases where you do need higher levels of scale or might run into merge conflicts with DVC. There's trade-offs though. DVC is definitely easier for a single developer / data scientist to get up and running w…
I think it's worth noting that DVC can be used to track artifacts that have been generated by other tools. For example, you could use MLFlow to run several model experiments, but at the end track the artifacts with DVC. Personally I think that this is the best way to use it.
However I agree that in general it's best for smaller projects and use cases. for example, it still shares the primary deficiency of Make in that it can only track files on the file system, and now things like ensuring a database table has been created (unless you 'touch' your own sentinel files).
Re: Data Version Control
#15The package phones home. One has to set an env var or fix several lines of code to prevent that.
I wondered how they'll make money https://www.crunchbase.com/organization/iterative-ai/company...
I think their plan was/is to make money on corporate licenses and support, as well as SaaS/cloud products.
Re: Data Version Control
#16Can it be used for large and fast changing datasets? Example: 100 TB, write us every 10 mins. Or, 1tb, parquet, 40% is rewritten daily.
DVC is expressly for tracking artifacts that are files on disk, and only by comparing their MD5 hashes. So it can definitely track the parquet files, but you are not going to get row or field diffs or anything like that.
Maybe Pachyderm or Dolt would be better tools here.
Re: Data Version Control
#17DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it…
> You are encouraged if you model your pipeline in DVC. Encouraged to do what? You might want to slow down on the use of parentheses, we are both getting lost in them.
I assume they meant to say "you are encouraged to use DVC to run your model and experiment pipeline". They want to encourage you to do this because they are trying to build a business around being a data science ops ecosystem. But the truth is that DVC is not a great tool for running "experiments" searching over a parameter space. it could be improved in that regard, but that's just not what I use it for nor is it what I recommend it to other people for.
However it's fantastic for tracking artifacts throughout an project that have been generated by other means, and for keeping those artifacts tightly in sync with Git, and for making it easy to share those artifacts without forcing people to re-run expensive pipelines.
Re: Data Version Control
#18DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it…
What’s best if parallel step processing is required?
Re: Data Version Control
#19If you just want a git for large data files, and your files don't get updated too often (e.g. an ML model deployed in production which gets updated every month) then git-lfs is a nice solution. Bitbucket and Github both have support for it.
Re: Data Version Control
#20Can it be used for large and fast changing datasets? Example: 100 TB, write us every 10 mins. Or, 1tb, parquet, 40% is rewritten daily.
DVC is expressly for tracking artifacts that are files on disk, and only by comparing their MD5 hashes. So it can definitely track the parquet files, but you are not going to get row or field diffs or anything like that. Maybe Pachyderm or Dolt would be better tools here.
Why would you use MD5 in anything written in the last 5 years? The SHA family is faster on modern hardware and there aren't trivial collisions floating around out there.