Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

Post reply on HN
Print view View original

11–20 of 142 posts

Re: Avoid Git LFS if possible

#11

Post by asymptosis »

Something missing from the list of problems: Git LFS is a http(s) protocol so is problematic at best when you are using Git over ssh[1].

The git-lfs devs obviously don't use ssh, so you get the feeling they are a bit exasperated by this call to support an industry standard protocol which is widely used as part of ecosystems and workflows involving Git.

[1] https://github.com/git-lfs/git-lfs/issues/1044

Re: Avoid Git LFS if possible

#13

Post by tux1968 »

goodcjw2 wrote: post #8

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?

Re: Avoid Git LFS if possible

#14

Post by temac »

If you just don't jump on random tech without good reasons, you already naturally apply this advice. Especially since once you really need it and also wants Git, there is not much alternative (as the author recognizes). In this context, just waiting for a potential "better support for handling of large files" of official Git makes little sense; plus I make the wild prediction that what will actually happen is that it's Git LFS that will (continue to) be improved and used by most people (and maybe even integrated in "official Git"?)

Re: Avoid Git LFS if possible

#15

Post by madjam002 »

As much as Git LFS is a bit of a pain, on recent projects I've resorted to committing my node_modules with Yarn 2 to Git using LFS and it works really well.

Note that with Yarn 2 you're committing .tar.gz's of packages rather than the JS files themselves, so it lends itself quite well to LFS as there are a smaller number of large files.

https://yarnpkg.com/features/zero-installs#how-do-you-reach-... https://yarnpkg.com/features/zero-installs#is-it-different-f...

Re: Avoid Git LFS if possible

#16

Post by slaymaker1907 »

Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.

Re: Avoid Git LFS if possible

#17

Post by snovv_crash »

rkangel wrote: post #9
dheera wrote: post #4

Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.

Some combination of the following two features: Partial clones ( https://docs.gitlab.com/ee/topics/git/partial_clone.html ) Shallow clones (see the --depth argument: https://linux.die.net/man/1/git-clone ) The problem with large files is not so much that putting a 1Gb file in Git is a problem. If you just have one revision of it, you get a 1Gb repo, and things run at a reasonable speed. The problem is when you have 1…

I found using git-lfs only in a subrepo worked well, since subrepos by default are checked out shallow.

Re: Avoid Git LFS if possible

#18

Post by toomuchtodo »

goodcjw2 wrote: post #8

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

Consider S3 storage and egress costs. You’re paying a flat rate to store and then pull that 50GB data (edit: removed an incorrect statement here).

Re: Avoid Git LFS if possible

#20

Post by Someone1234 »

toomuchtodo wrote: post #18
goodcjw2 wrote: post #8

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

Consider S3 storage and egress costs. You’re paying a flat rate to store and then pull that 50GB data (edit: removed an incorrect statement here).

Just for clarity, it is 50 GB of storage and 50 GB of bandwidth.

So definitely not "as much as you want." If you pull it too many times you may get charged another $5.

Post reply on HN