Avoid Git LFS if possible
gregoryszorc.com
Avoid Git LFS if possible
1–10 of 142 posts
Re: Avoid Git LFS if possible
#2My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s):
1. Type information. Enough to synthesize a fake example.
2. A simple preview. This can be a thumb or video snippet, for example.
3. Checksum and URL of the big file.
This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time.
You can then also use the best version control tool for the job for the particular big files in question.
Re: Avoid Git LFS if possible
#3My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You…
Is this just a manual equivalent of git LFS, or is there some advantage here?
Re: Avoid Git LFS if possible
#4Okay, so I should avoid it. What is the alternative?
I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS.
LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.
Re: Avoid Git LFS if possible
#5Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.
According to the article you should use mercurial or PlasticSCM because otherwise you might have to rewrite your history to get to some hypothetical git solution that isn't even on the roadmap.
I think I'll stick to LFS.
Re: Avoid Git LFS if possible
#6Yep. All of this. I tried using Git LFS for a project and reverted back to links to cloud server for the large binary blobs and hashes on those blobs.
Re: Avoid Git LFS if possible
#7Pushing Github past the 100mb limit has to be the most requested feature. Ridiculous that we have to use the fudge that is GitLFS.
It just adds complication for a limit that shouldn't be there anyway.
Re: Avoid Git LFS if possible
#8A side topic: is there a concrete reason why github's LFS solution has to be so expensive?
IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.
Re: Avoid Git LFS if possible
#9Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.
Some combination of the following two features:
Partial clones (https://docs.gitlab.com/ee/topics/git/partial_clone.html)
Shallow clones (see the --depth argument: https://linux.die.net/man/1/git-clone)
The problem with large files is not so much that putting a 1Gb file in Git is a problem. If you just have one revision of it, you get a 1Gb repo, and things run at a reasonable speed. The problem is when you have 10 revisions of the 1Gb file and you end up dealing with 10Gb of data when you only want one, because the default git clone model is to give you the full history of everything since the beginning of time. This is fine for (compressible) text files, less fine for large binary blobs.
Git-lfs is a hack and it has caused me pain every time I've used it, despite Gitlab having good support for it. Some of this is more implementation detail - the command line UI has some wierdness to it, there's no clear error if someone doesn't have git-lfs when cloning and so something in your build process down the line breaks with a weird error because you've got a marker file instead of the expected binary blob. Some of it is inherent though - the hardest problem is that we now can't easily mirror the git repo from our internal gitlab to the client's gitlab because the config has to hold the http server address with the blobs in. We have workarounds but they're not fun.
The solution is to get over the 'always have the whole repository' thing. This is also useful for massive monorepos because you can clone and checkout just the subfolder you need and not all of everything.
I say this, but I haven't yet used partial clones in anger (unlike git-lfs). I have high hopes though, and it's a feature in early days.
Re: Avoid Git LFS if possible
#10All three points are really just the same point repeated three times: That it isn't part of core/official GIT ("stop gap" until official, irreversible to later official solution, and adds complexity that an official version would lack due to extra/third party tooling).
I'm frankly surprised GIT hasn't made LFS an official part by now. It fixes the problem, the problem is common and real, and GIT hasn't offered a better alternative.
If LFS was made official it would solve this critique, since that is really the only critique here.