Data Version Control

Post by [deleted] » Tue, Oct 04, 2022, 10:27 PM UTC

kernelsanderz wrote: post #36

adhocmobility wrote: post #19

If you just want a git for large data files, and your files don't get updated too often (e.g. an ML model deployed in production which gets updated every month) then git-lfs is a nice solution. Bitbucket and Github both have support for it.

I do feel like git-lfs is a good solution. Once you have 10s or 100s of GB of files (eg. a computer vision project), this gets pretty pricey. Ideally I'd love to use git-lfs on top of S3, directly. I've looked into git-annex and various git-lfs proxies, but I'm not sure they're maintained well enough to be trusting it with long-term data storage. Huggingface datasets are built on git-lfs and it works really well for…

Post by davidatbu » Fri, Oct 07, 2022, 1:00 PM UTC

smeagull wrote: post #58

davidatbu wrote: post #56

Earlier quoted context omitted.

How do you merge multiple versions of data using tensorboard? Or what other tool handles that for you? What's the case for handling code and data separately? In my experience, the primary motivation for using such a tool are easy reproducibility through easy tracking of code, hyperparams, and data. It's not obvious to me how that goal would be advanced by tracking code and data separately.

Tensorboard doesn't do that, I was referring to things a dataset/model management tool should do. For us, Tensorboard tracks the datasets as hyperparams. The actual multiple versions of data end up being handled on the warehouse side. Prefect is what we use for running those DAGs to make the different versions. Handling code and data separately is important, to allow easy updates to one or the other. They are loosely…