Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

Post reply on HN
Print view View original

31–40 of 142 posts

Re: Avoid Git LFS if possible

#31

Post by alkonaut »

asymptosis wrote: post #11

Something missing from the list of problems: Git LFS is a http(s) protocol so is problematic at best when you are using Git over ssh[1]. The git-lfs devs obviously don't use ssh, so you get the feeling they are a bit exasperated by this call to support an industry standard protocol which is widely used as part of ecosystems and workflows involving Git. [1] https://github.com/git-lfs/git-lfs/issues/1044

That issue has come a long way though, already a draft PR! This seems like it could actually happen.

Re: Avoid Git LFS if possible

#32

Post by Game_Ender »

The latest version of git has a very similar feature called “partial clones” to what the author describes for Mercurial. All the data is still in your history, no extra tools are needed, but you only fetch the blobs from the server for the commits you checkout. So just like LFS larger blobs not on master are effectively free, but you still grab all the blobs for your current commit.

You need server side support, which GitHub and GitLab have, and then a special clone command:

    git clone --filter=blob:none
Some background about the feature is here: https://github.blog/2020-12-21-get-up-to-speed-with-partial-...

Re: Avoid Git LFS if possible

#33

Post by alkonaut »

slaymaker1907 wrote: post #16

Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.

The problem I see is that things like commit hashes which are etched in history in bug reports, version tags etc, instantly lose meaning. Whether or not that’s a problem depends on how much of that you have.

Re: Avoid Git LFS if possible

#34

Post by TeeMassive »

The reason the author provides is in my opinion weak compared to both his alternatives.

Sure, lfs contaminates a repository, so do large files, sensitive data removal, and references to packages and package managers that might become obsolete or non-existent in the future. The chance of your project compiling after 15 years, the age of git by the way, are very slim, and the chance that having a entirely compilable history being useful even slimmer.

And I think the author's statement about setupping up lfs being hard is exaggerated. It's a handful of command lines that should be in the "welcome at our company" manual anyway.

I've used lfs in the past and while it can be misused, as with all other tools, it does the job without too much headaches compared submodules and ignored tracked files.

Re: Avoid Git LFS if possible

#35

Post by alkonaut »

dheera wrote: post #4

Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.

If you are like most people you use systems that speak git (from Microsoft, Jetbrains, GitHub, Atlassian…) but rarely or less fluently anything else so the problem I’m trying to solve isn’t “which VCS lets me work well with large files” but rather “I’m stuck with git so what do I do with my large files”.

Your option is basically Git LFS, possibly also VFSForGit, or putting your large files in separate storage.

Re: Avoid Git LFS if possible

#36

Post by sjansen »

slaymaker1907 wrote: post #16

Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.

The issue is breaking external references.

Do you include git SHAs in your bug tracking system? Or perhaps your department wiki links to a specific commit to document lessons learned? Maybe you're using Sentry and find including the git SHA of the build to be invaluable for troubleshooting?

For some organizations, rewriting history would be a non-event and for others it would be a major disruption.

Re: Avoid Git LFS if possible

#37

Post by wbillingsley »

The solution I've tended to use in classes (where there'll always be some student who hasn't installed LFS) is to store the large files in Artifactory, so they are pulled in at build-time in the same way as libraries.

This seemed to me a sensible approach as Artifactory is a repository for binaries (usually, the compiled output of a project). It also seemed to me that the decisions on which versions to retain and when an update to a binary is expected or when that resource is now frozen and a replacement would be a new version is similar to the decision on when a build is a snapshot vs a release.

Re: Avoid Git LFS if possible

#38

Post by klodolph »

Someone1234 wrote: post #10

All three points are really just the same point repeated three times: That it isn't part of core/official GIT ("stop gap" until official, irreversible to later official solution, and adds complexity that an official version would lack due to extra/third party tooling). I'm frankly surprised GIT hasn't made LFS an official part by now. It fixes the problem, the problem is common and real, and GIT hasn't offered a bett…

> All three points are really just the same point repeated three times

Absolutely not. Having worked with Mercurial LFS and Git LFS, the differences seem subtle but they are there. Basically,

In Mercurial, LFS is (to an extent) an implementation detail of how you check out a repository. It doesn't mean altering the repository contents itself (the data), it just means altering how you get that data. Contrast with Git LFS, where the data itself must be altered in order to become LFS data, and the "LFS flag" is recorded in history.

This is not something that you would solve by upstreaming LFS. You would need to redesign LFS.

Re: Avoid Git LFS if possible

#39

Post by coley »

CreepGin wrote: post #19

I've been using Git LFS with several large Unity projects in the past several years. Never really had any problems. It was always just "enable and forget" kind of thing.

This is my experience as well so far. It took 15-20 minutes to learn about it, install it, and setup configs. Since then I haven't had to think about it once.

Re: Avoid Git LFS if possible

#40

Post by klodolph »

hpcjoe wrote: post #23

Just this past week, git lfs was throwing smudge errors for me. Not really sure what the issue was, I followed the recommendations to disable, pull, and re-enable. And got them again. So I disabled. And left it disabled. Not a solution. This said, the whole git-lfs bit feels like a (bad) afterthought the way its implemented. I'd love to see some significant reduction of complexity (you shouldn't need to do 'git lfs e…

I would say that if you care about good LFS support, that is a sufficient reason to use Mercurial. Harder to find Mercurial hosting these days, though, but I'm not worried that the Mercurial project will die off (since both Facebook and Google use it, in some manner).
Post reply on HN