Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

41–50 of 142 posts

Re: Avoid Git LFS if possible

#41

> Git LFS is a Stop Gap Solution Build the real thing then..

The author of the article is a Mercurial maintainer, and Mercurial has the "real thing" implemented already (and it has been part of Mercurial since at least 2012, at least in some form). So it's already done, just not for Git.

Re: Avoid Git LFS if possible

#42
post #13

Earlier quoted context omitted.

$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?

It's a very large markup on the small-user retail cost of the basic thing they're providing (web-accessible, access-controlled file storage—see, for example, BackBlaze B2) but that's utterly typical of services that can get away with charging you a "convenience fee" for that sort of thing once you're on their SaaS. 2-3x markup isn't unusual, and that's about what this is, and that's above typical retail—even if GH's…

Yes, I would estimate that the markup is more like 5-10x for GitHub LFS.

Re: Avoid Git LFS if possible

#44
post #2

My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You…

Is this just a manual equivalent of git LFS, or is there some advantage here?

It's a design pattern that ensures testability of the system without any dependencies on the big files.

Re: Avoid Git LFS if possible

#45
post #36

Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.

The issue is breaking external references. Do you include git SHAs in your bug tracking system? Or perhaps your department wiki links to a specific commit to document lessons learned? Maybe you're using Sentry and find including the git SHA of the build to be invaluable for troubleshooting? For some organizations, rewriting history would be a non-event and for others it would be a major disruption.

Yeah, git is really not a mature or well-designed VCS. The fact that you can trivially lose the supposed permanent reference -- and that it's encouraged as part of several common workflows at that -- should be more than enough to demonstrate this. If you care about history, use a VCS like Fossil.

Re: Avoid Git LFS if possible

#46

Earlier quoted context omitted.

It's a very large markup on the small-user retail cost of the basic thing they're providing (web-accessible, access-controlled file storage—see, for example, BackBlaze B2) but that's utterly typical of services that can get away with charging you a "convenience fee" for that sort of thing once you're on their SaaS. 2-3x markup isn't unusual, and that's about what this is, and that's above typical retail—even if GH's…

Yes, I would estimate that the markup is more like 5-10x for GitHub LFS.

Exactly. For us we are dealing with lots of large gaming assets and those are burning through those $50 data packs like butters :)

Re: Avoid Git LFS if possible

#47

The latest version of git has a very similar feature called “partial clones” to what the author describes for Mercurial. All the data is still in your history, no extra tools are needed, but you only fetch the blobs from the server for the commits you checkout. So just like LFS larger blobs not on master are effectively free, but you still grab all the blobs for your current commit. You need server side support, whic…

This looks too aggressive. The nice thing about git-lfs is that only the binary file type(s) you care about are run through git-lfs. All other ordinary diffable text is treated normally.

The blobless clone is going to be ensaddening the next time that I'm examining the history of some source code when I'm hacking away without a network connection.

Re: Avoid Git LFS if possible

#48
Did he really just try to make the argument that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future?

LFS has existed for several years, and as far as I know Git still doesn’t have support for large files. At this point I’m not holding out much hope.

Re: Avoid Git LFS if possible

#49
This opinion only lists issues, not solutions. Sure, they advertise mercurial, but migrating from git to mercurial is unrealistic for many cases.

I'd title it: "Why Mercurial is better than git+LFS"

Re: Avoid Git LFS if possible

#50

The latest version of git has a very similar feature called “partial clones” to what the author describes for Mercurial. All the data is still in your history, no extra tools are needed, but you only fetch the blobs from the server for the commits you checkout. So just like LFS larger blobs not on master are effectively free, but you still grab all the blobs for your current commit. You need server side support, whic…

This looks too aggressive. The nice thing about git-lfs is that only the binary file type(s) you care about are run through git-lfs. All other ordinary diffable text is treated normally. The blobless clone is going to be ensaddening the next time that I'm examining the history of some source code when I'm hacking away without a network connection.

You can mitigate a bit of this by only ignoring blobs over a certain size like "--filter=blob:limit=256k" which should allow most ordinary text files through.

In the end it's the same as LFS though in that without a network examining old commits without a network is a bummer. No free lunch here besides something a bit more complex like git-annex.

Post reply on HN