91–100 of 142 posts
Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.
Did he really just try to make the argument that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? LFS has existed for several years, and as far as I know Git still doesn’t have support for large files. At this point I’m not holding out much hope.
git supports large files, it just can't track changes in binary files efficiently and if they're large you check in a new blob every modification. If they're just sitting around it's fine, but then why would you have them in VC
I despise LFS. I’m sure that if you know how to use it... maybe... you can figure it out. That said; here’s my battle story: Estimate the time it’ll take to move all our repositories from a to b they said. Us: with all branches? Them: just main and develop. Us: you just clone and push to the new origin, it’s not zero but it’s trivial. Weeks later... Yeah. LFS is now banned. LFS is not a distributed version control sy…
Earlier quoted context omitted.
The issue is breaking external references. Do you include git SHAs in your bug tracking system? Or perhaps your department wiki links to a specific commit to document lessons learned? Maybe you're using Sentry and find including the git SHA of the build to be invaluable for troubleshooting? For some organizations, rewriting history would be a non-event and for others it would be a major disruption.
Yeah, git is really not a mature or well-designed VCS. The fact that you can trivially lose the supposed permanent reference -- and that it's encouraged as part of several common workflows at that -- should be more than enough to demonstrate this. If you care about history, use a VCS like Fossil.
A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.
Yea i actually wrote my own file chunking and general git-lfs-like backend for this exact reason. I liked Git LFS, but Github's pricing felt insane for my indie dev. For my needs i could backup onto a local server, network drive, or w/e at an insanely cheaper price. Hell even uploading to an S3 compatible API was insanely cheaper than Github. That and i really hated the feeling that Git LFS was being designed for a s…
Earlier quoted context omitted.
Sort of. The way that the author talks about Mercurial as not having this problem makes me think they're talking about something related but subtly different. In particular, AFAICT, Mercurial requires the exact same thing as what you're pointing out. If you want to completely disable use of largefiles then you still have to run `hg lfconvert` at some point. That also changes your revision history. The "one-way door"…
I don’t think it’s clear but mercurial has two solutions for large file support. The original “largefiles” which had all the same designs and issues as Git LFS they bring up in the blog post, and “lfs” which is newer. I’ve used largefiles and ran into these issues and ended up having to turn it off after a few years because it’s so problematic with the tooling since it modifies the underlying mercurial commit structu…
However, my impression is that in fact largefiles is basically the only game in town and Mercurial LFS if anything is meant to be even more like Git LFS to the point of being compatible with it.
The thing I'm more curious about is I don't immediately see how large file support in git (or mercurial), whether implemented as a separate tool or natively, could ever feasibly be "transparently erasable," that is rewindable back to be absolutely identical to a repository with no large files support without rewriting revision history.
It doesn't seem impossible (e.g. maybe you could somehow maintain a duplicate shadow revision history and transparently intercept syscalls?), but the approaches I can think of all have pretty hefty downsides and feel even more like hacks than the current crop of tools.
Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.
To me, most cases of large files in VCS seem like using a hammer as a screwdriver.
Earlier quoted context omitted.
git supports large files, it just can't track changes in binary files efficiently and if they're large you check in a new blob every modification. If they're just sitting around it's fine, but then why would you have them in VC
It tracks the changes fine. It’s just that it doesn’t make sense to track changes in a binary.
LRzip happens to have such a format preprocessor that would make for exceedingly efficient binary history at cost of being more similar to git pack file than incremental versions.
Then again, GitHub in particular sets a very low limit on binary size in version control.
I despise LFS. I’m sure that if you know how to use it... maybe... you can figure it out. That said; here’s my battle story: Estimate the time it’ll take to move all our repositories from a to b they said. Us: with all branches? Them: just main and develop. Us: you just clone and push to the new origin, it’s not zero but it’s trivial. Weeks later... Yeah. LFS is now banned. LFS is not a distributed version control sy…
What is your alternative then? Version control binary files and have your repos grow gigabytes?
If you're not rewriting the files, also yes.
If you don't need the history, put them on a normal web server.
Earlier quoted context omitted.
git supports large files, it just can't track changes in binary files efficiently and if they're large you check in a new blob every modification. If they're just sitting around it's fine, but then why would you have them in VC
It tracks the changes fine. It’s just that it doesn’t make sense to track changes in a binary.
Tracking changes of binaries makes a lot of sense if you use that to only store incremental changes to the file. Git stores each modification of a binary file as a separate blob since it doesn't know how to track its changes.
This is mitigated in large parts by the compression applied in git-gc, after packed, objects went from 196mb to 108mb.