Data Version Control • Hacker News Board

Re: Data Version Control

#51

Post by shicholas » Mon, Oct 21, 2024, 9:24 PM UTC

dijksterhuis wrote: post #35

shicholas wrote: post #4

What are the benefits of DVC over Apache Iceberg? If anyone used both, I'd be curious about your take. Thanks!

head is a bit discombobulated today, but i’ll give this a shot when i say ‘blob’ data, a good example to think of is a set of really long 1080p video files. tl;dr version * throw data into dvc when unstructured ‘blob’ data. * throw it into iceberg when you’ve got structured data. benefits of dvc over iceberg: * not forcing ‘blob’ data into a tabular format and all the “fun” (read: annoying) processing steps that come…

This is great thank you!

Re: Data Version Control

#52

Post by jendefig » Mon, Oct 21, 2024, 9:47 PM UTC

tomtom1337 wrote: post #30

The animated ripple across the «what’s new» button is infuriating. It keeps drawing my attention from reading what this is.

Thank you for your feedback! We can see how this could be distracting. I'm the Community Manager, this has been fixed!

Re: Data Version Control

#53

Post by bach4ants » Sat, Oct 26, 2024, 12:20 PM UTC

bramathon wrote: post #6

I've used DVC for most of my projects for the past five years. The good things is that it works a lot like git. If your scientists understand branches, commits and diffs, they should be able to understand DVC. The bad thing is that it works like git. Scientists often do not, in fact, understand or use branches, commits and diffs. The best thing is that it essentially forces you to follow Ten Simple Rules for Reproduc…

I have noticed this as well. There is a huge resistance to learning Git, and I think it's partly warranted. Researchers know what it is, and know that it's valuable, but think it will take too long to learn and they want to move fast. I recently started building a tool called Calkit (https://github.com/calkit/calkit) in an attempt to simply and unify Git and DVC for these types of researchers. Hoping to convince folks that working reproducibly is actually faster in the long run, never mind the fact that it makes their work more directly usable for pushing the field forward more quickly overall.

Re: Data Version Control

#54

Post by bach4ants » Sat, Oct 26, 2024, 12:33 PM UTC

stochastastic wrote: post #46

amelius wrote: post #39

Earlier quoted context omitted.

Sounds like it is more a framework than a tool. Not everybody wants a framework.

It doesn’t force you to use any of the extra functionality. My team has been using it just for the version control part for a couple years and it has worked great.

Yep. I personally like DVC's pipeline implementation because it's lightweight and language-agnostic, but haven't gotten into using their experiment tracking features.