Data Version Control
dvc.org
Data Version Control
1–10 of 63 posts
Re: Data Version Control
#2The package phones home. One has to set an env var or fix several lines of code to prevent that.
Re: Data Version Control
#3Can it be used for large and fast changing datasets?
Example: 100 TB, write us every 10 mins.
Or, 1tb, parquet, 40% is rewritten daily.
Re: Data Version Control
#4Can it be used for large and fast changing datasets? Example: 100 TB, write us every 10 mins. Or, 1tb, parquet, 40% is rewritten daily.
What about Apache Iceberg for those?
Re: Data Version Control
#5The package phones home. One has to set an env var or fix several lines of code to prevent that.
[deleted]
Re: Data Version Control
#6The package phones home. One has to set an env var or fix several lines of code to prevent that.
I wondered how they'll make money
https://www.crunchbase.com/organization/iterative-ai/company...
Re: Data Version Control
#7The package phones home. One has to set an env var or fix several lines of code to prevent that.
I wondered how they'll make money https://www.crunchbase.com/organization/iterative-ai/company...
They won't, they can make investor money back only from selling company to Amazon/Microsoft/Google but in this economy it won't happen.
Re: Data Version Control
#8DVC has had the following problems, when I tested it (half a year ago):
I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it.
You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it takes a lot of time to run a pipeline while you are on a beefy machine and only one core is used. Obviously, you cannot run other tools (e.g. snakemake) to distribute/parallelize on multiple machines. Running one (part of a) stage has also some overhead, because it does commit/checks after/before running the executable of the task.
Sometimes you get merge conflicts, if you run a (partial parmaretized) stage on one machine and the other part on the other machine manually. These are cumbersome to fix.
Currently, I think they are more focused on ML features like experiment tracking (I prefer other mature tools here) instead of performance and data safety.
There is an alternative implementation from a single developer (I cannot find it right now) that fixes some problems. However, I do not use this because it propably will not have the same development progress and testing as DVC.
This sounds negative but I think it is currently the one of the best tools in this space.
Re: Data Version Control
#9Can anyone compare this to DataLad [1], which someone introduced to me as "git for data"?
Re: Data Version Control
#10DVC has had the following problems, when I tested it (half a year ago): I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it. You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it…
> You are encouraged if you model your pipeline in DVC.
Encouraged to do what?
You might want to slow down on the use of parentheses, we are both getting lost in them.