Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Version Control

dvc.org

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Data Version Control

#21

Post by mkbehbehani »

causal wrote: post #17

This useful for large binaries?

Yes, I’ve been using it for about a year to populate databases with a reference DB dump. The current file is about 18GB. I use cloudflare R2 as the backing store so even though it’s being pulled very frequently the cloudflare bill is a few bucks per month.

Re: Data Version Control

#22

Post by dmd »

foobarbecue wrote: post #13
andrew_lettuce wrote: post #7

Earlier quoted context omitted.

I don't see any real benefits, as it feels like using the tool you already know even though it's not quite right. Iceberg is maybe geared towards slower changing models than this approach?

username checks out

You must be into Apache Ignite.

Re: Data Version Control

#23

Post by thangngoc89 »

natsucks wrote: post #20
causal wrote: post #17

This useful for large binaries?

Would appreciate a good answer to this question. I deal with large medical imaging data (DICOM) and i cannot tell whether it's worth it and/or feasible.

It's very much feasible. I'm currently using DVC for DICOM, the repo has growth to about 5TB of small dcm files (less than < 100KB each). We use a NFS mounted NAS for development but the DVC's cache needs to be on the NVMe, otherwise performance would be terrible.

Re: Data Version Control

#24

Post by thangngoc89 »

azinman2 wrote: post #19
miki123211 wrote: post #15

Earlier quoted context omitted.

DVC does a lot more than git. It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically. There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the i…

So where do the adjusted 10M rows live instead? S3?

DVC support multiple remotes. S3 is one of them, there are also WebDAV, local FS, Google Drive, and a bunch of others. You could see the full list here [0]. Disclaimer: not affiliated with DVC in anyway, just a user.

[0] https://dvc.org/doc/user-guide/data-management/remote-storag...

Re: Data Version Control

#26

Post by woodson »

notrealyme123 wrote: post #25

I had a lot of problems when using it with a dataset of many jpg Files. The indexing for every dvc status took many minutes to check every file. Caching did not work. Sadly I had to let go of it.

Yes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). For large sharded datasets (e.g. WebDataset), you're better off with other solutions, especially when your ML pipeline can stream them directly from object storage.

Re: Data Version Control

#28

Post by dmpetrov »

woodson wrote: post #26
notrealyme123 wrote: post #25

I had a lot of problems when using it with a dataset of many jpg Files. The indexing for every dvc status took many minutes to check every file. Caching did not work. Sadly I had to let go of it.

Yes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). For large sharded datasets (e.g. WebDataset), you're better off with other solutions, especially when your ML pipeline can stream them directly from object storage.

Right, DVC caches data for consistency and reproducibility.

If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset and can stream from tar archives and filter images by metadata.

WebDataset example: https://github.com/iterative/datachain/blob/main/examples/mu...

Re: Data Version Control

#29

Post by ulnarkressty »

We actually were considering DVC, however for our particular use case (huge video files which don't change much) the git paradigm was not that useful - you need at least one copy of the data on the origin and another one on each system that's doing the training. So in the end we just went with files and folders on a NAS, seemed to work good enough.

A hybrid solution of keeping dataset metadata under DVC and then versioning that could work. This was many years ago though and I would be curious if there are any other on-prem data versioning solutions, when I last searched all of them seem geared towards the cloud.

Post reply on HN