The animated ripple across the «what’s new» button is infuriating. It keeps drawing my attention from reading what this is.
I bound Ctrl-Shift-Z to Ublock Origin's Zapper mode. It's really helpful
Data Version Control
41–50 of 54 posts
Re: Data Version Control
#42How does it compare to Oxen? https://github.com/Oxen-AI/Oxen
Maintainer of Oxen here, we initially built Oxen because DVC was pretty painfully slow to work with, and had a lot of extra bells and whistles that we didn’t need. Under the hood we optimized the merkle tree structure, hashing algorithms, network protocols, etc to make it speedy when it came to large datasets. We have a pretty nice front end at https://oxen.ai for viewing and querying the data as well.
Happy to answer any thoughts or questions!
Re: Data Version Control
#43Earlier quoted context omitted.
DVC does a lot more than git. It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically. There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the i…
Sounds like it is more a framework than a tool. Not everybody wants a framework.
> Sounds like it is more a framework than a tool.
> Not everybody wants a framework.
The second part of this comment seems strange to me. Surely nothing on Hacker News is shared with the expectation that it will be interesting, or useful, to everyone. Equally, surely there are some people on HN who will be interested in a framework, even if it might be too heavy for other people.
Re: Data Version Control
#44Earlier quoted context omitted.
Sounds like it is more a framework than a tool. Not everybody wants a framework.
> Sounds like it is more a framework than a tool. > Not everybody wants a framework. The second part of this comment seems strange to me. Surely nothing on Hacker News is shared with the expectation that it will be interesting, or useful, to everyone . Equally, surely there are some people on HN who will be interested in a framework, even if it might be too heavy for other people.
Just saying that what makes Git so appealing is that it does one thing well, and from this view DVC seems to be in an entirely different category.
Re: Data Version Control
#45How does it compare to Oxen? https://github.com/Oxen-AI/Oxen
Can this be used with GitHub?
If yes, I would shift from dvc immediately
Re: Data Version Control
#46Earlier quoted context omitted.
DVC does a lot more than git. It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically. There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the i…
Sounds like it is more a framework than a tool. Not everybody wants a framework.
It doesn’t force you to use any of the extra functionality. My team has been using it just for the version control part for a couple years and it has worked great.
Re: Data Version Control
#47hi there! Maintainer and author here. Excited to see DVC on the front page! Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.
Mostly consult as a data engineer not ML ops but I’m interested in some aspects of this. We have 10 years of parquet files from 300+ different kafka topic and we’re currently migrating to apache iceberg. We’ll back fill on a need only basis and it would be nice to track that with git. Would this be a good fit for that?
Another potential aspect would be tracking schema evolution in a nicer way than we currently do.
thx in advance, huge fan of anything-as-code and think it’s a great fit for data (20+ years in this area).
Re: Data Version Control
#48hi there! Maintainer and author here. Excited to see DVC on the front page! Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.
Thanks for making and sharing DVC! It’s been a big help.
Is there any support that would be helpful? I’ll look at the project page too.
Re: Data Version Control
#49hi there! Maintainer and author here. Excited to see DVC on the front page! Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.
Thanks for making and sharing DVC! It’s been a big help. Is there any support that would be helpful? I’ll look at the project page too.
Thank you!
Just shoot an email to support and mention HN. I’ll read and reply.
Re: Data Version Control
#50Earlier quoted context omitted.
Yes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). For large sharded datasets (e.g. WebDataset), you're better off with other solutions, especially when your ML pipeline can stream them directly from object storage.
Right, DVC caches data for consistency and reproducibility. If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset and can stream from tar archives and filter images by metadata. WebDataset example: https://github.com/iterative/datachain/blob/main/examples/mu...
Thank you! Thats news to me. I will absolutely give it a try