Avoid Git LFS if possible • Hacker News Board

Re: Avoid Git LFS if possible

#21

Post by adkadskhj » Wed, May 12, 2021, 10:17 PM UTC

goodcjw2 wrote: post #8

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

Yea i actually wrote my own file chunking and general git-lfs-like backend for this exact reason. I liked Git LFS, but Github's pricing felt insane for my indie dev. For my needs i could backup onto a local server, network drive, or w/e at an insanely cheaper price.

Hell even uploading to an S3 compatible API was insanely cheaper than Github.

That and i really hated the feeling that Git LFS was being designed for a server architecture. I didn't have an easy way to locally dump the written files without running an HTTP server.

There are a couple Git LFS servers that upload to, say, S3 - but i really just wanted a dumb FS or SSH dump of my LFS files. Running a localhost server feels so.. anti-Git to me.

Re: Avoid Git LFS if possible

#22

Post by nerdponx » Wed, May 12, 2021, 10:17 PM UTC

dheera wrote: post #4

Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.

DVC [0] is great for data science applications, but I don't see why you couldn't use it as a general-purpose LFS replacement.

It doesn't fix all of the problems with LFS, but it helps a lot with some of them (and happens to also be a decent Make replacement in certain situations).

[0]: https://dvc.org/

Re: Avoid Git LFS if possible

#23

Post by hpcjoe » Wed, May 12, 2021, 10:21 PM UTC

Just this past week, git lfs was throwing smudge errors for me. Not really sure what the issue was, I followed the recommendations to disable, pull, and re-enable. And got them again. So I disabled. And left it disabled.

Not a solution.

This said, the whole git-lfs bit feels like a (bad) afterthought the way its implemented. I'd love to see some significant reduction of complexity (you shouldn't need to do 'git lfs enable', it should be done automatically), and increases in resiliency (sharding into FEC'ed blocks with distributed checksums, etc.) so we don't have to deal with 'bad' files.

I was a fan of mercurial before I switched to git ... it was IMO an easier/better system at the time (early 2010s). Not likely to switch now though.

Re: Avoid Git LFS if possible

#24

Post by dmm » Wed, May 12, 2021, 10:21 PM UTC

breck wrote: post #2

My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You…

Tools like git-annex or dvc support similar strategies.

Re: Avoid Git LFS if possible

#25

Post by moshmosh » Wed, May 12, 2021, 10:30 PM UTC

tux1968 wrote: post #13

goodcjw2 wrote: post #8

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?

It's a very large markup on the small-user retail cost of the basic thing they're providing (web-accessible, access-controlled file storage—see, for example, BackBlaze B2) but that's utterly typical of services that can get away with charging you a "convenience fee" for that sort of thing once you're on their SaaS. 2-3x markup isn't unusual, and that's about what this is, and that's above typical retail—even if GH's not managing the storage and such themselves, they're likely getting an even better (bulk) rate.

Re: Avoid Git LFS if possible

#26

Post by dwohnitmok » Wed, May 12, 2021, 10:32 PM UTC

git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you.

You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3).

The main issue that git-annex does not solve is that, like Git LFS, it's not a part of git proper and it shows in its occasionally clunky integration. By virtue of having more knobs and dials it also potentially has more to learn than Git LFS.

Re: Avoid Git LFS if possible

#27

Post by toomuchtodo » Wed, May 12, 2021, 10:33 PM UTC

Someone1234 wrote: post #20

toomuchtodo wrote: post #18

Earlier quoted context omitted.

Consider S3 storage and egress costs. You’re paying a flat rate to store and then pull that 50GB data (edit: removed an incorrect statement here).

Just for clarity, it is 50 GB of storage and 50 GB of bandwidth. So definitely not "as much as you want." If you pull it too many times you may get charged another $5.

Thanks for the correction!

Re: Avoid Git LFS if possible

#28

Post by korijn » Wed, May 12, 2021, 10:39 PM UTC

I'm honestly super content with LFS. Wrote our own little API server to hook it up to Azure Blob Storage, never have issues with it. I don't recognize the issues mentioned in the article at all. Our whole team relies on it for years, and it delivers. No problems. Keep up the great work, git-lfs maintainers! Much love.

Re: Avoid Git LFS if possible

#29

Post by alkonaut » Wed, May 12, 2021, 10:42 PM UTC

I’m using Git+LFS because my issue tracker, CI/CD etc natively speaks it. Not because it’s in any way superior or even on par with the large file handling of Mercurial (or even SVN to be honest).

Re: Avoid Git LFS if possible

#30

Post by viraptor » Wed, May 12, 2021, 10:42 PM UTC

robmsmt wrote: post #7

Pushing Github past the 100mb limit has to be the most requested feature. Ridiculous that we have to use the fudge that is GitLFS. It just adds complication for a limit that shouldn't be there anyway.

You can use gitlab instead with a few GB limit instead.