Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

Post reply on HN
Print view View original

51–60 of 142 posts

Re: Avoid Git LFS if possible

#51

Post by jfim »

klodolph wrote: post #40
hpcjoe wrote: post #23

Just this past week, git lfs was throwing smudge errors for me. Not really sure what the issue was, I followed the recommendations to disable, pull, and re-enable. And got them again. So I disabled. And left it disabled. Not a solution. This said, the whole git-lfs bit feels like a (bad) afterthought the way its implemented. I'd love to see some significant reduction of complexity (you shouldn't need to do 'git lfs e…

I would say that if you care about good LFS support, that is a sufficient reason to use Mercurial. Harder to find Mercurial hosting these days, though, but I'm not worried that the Mercurial project will die off (since both Facebook and Google use it, in some manner).

Is Facebook still using mercurial? It seems that there was a blog post about it in 2014, but their repo[0] just seems to say that their codebase was originally based on/evolved from mercurial.

[0] https://github.com/facebookexperimental/eden

Re: Avoid Git LFS if possible

#52

Post by wokwokwok »

I despise LFS.

I’m sure that if you know how to use it... maybe... you can figure it out.

That said; here’s my battle story:

Estimate the time it’ll take to move all our repositories from a to b they said.

Us: with all branches?

Them: just main and develop.

Us: you just clone and push to the new origin, it’s not zero but it’s trivial.

Weeks later...

Yeah. LFS is now banned.

LFS is not a distributed version control system; once you use it, a clone is no longer “as good” as the original, because it refers to a LFS server that is independent of your clone.

...also, actually cloning all the LFS content from git lab is both slow and occasionally broken in a way that requires you to restart the clone.

:(

Re: Avoid Git LFS if possible

#53

Post by cesarb »

dwohnitmok wrote: post #26

git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you. You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3). The main issue that git-annex does not solve is that, like Git LFS,…

With git-annex you have the same "one-way door" behavior: it replaces large files with a pointer to the content (in git-annex, it's a relative symbolic link which by default encodes the real file's size and hash), which is stored in git-annex's own database.

Re: Avoid Git LFS if possible

#54

Post by remram »

cesarb wrote: post #53
dwohnitmok wrote: post #26

git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you. You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3). The main issue that git-annex does not solve is that, like Git LFS,…

With git-annex you have the same "one-way door" behavior: it replaces large files with a pointer to the content (in git-annex, it's a relative symbolic link which by default encodes the real file's size and hash), which is stored in git-annex's own database.

That content can easily be moved in bulk though. It is true that you have to use git-annex command to do so, but this is different from LFS where the complete set of historical files is only stored on the server and can't be moved at all.

edit: The article claims it's a "one-way door" because you can't move to an altogether different system without rewriting history, which is true of git-annex. My bad.

Re: Avoid Git LFS if possible

#55

Post by remram »

Someone1234 wrote: post #20
toomuchtodo wrote: post #18

Earlier quoted context omitted.

Consider S3 storage and egress costs. You’re paying a flat rate to store and then pull that 50GB data (edit: removed an incorrect statement here).

Just for clarity, it is 50 GB of storage and 50 GB of bandwidth. So definitely not "as much as you want." If you pull it too many times you may get charged another $5.

For S3, that's $1.15 storage ($0.023/GB) + $4.50 transfer ($0.09/GB) so it's actually very comparable.

Re: Avoid Git LFS if possible

#56

Post by remram »

dmm wrote: post #24
breck wrote: post #2

My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You…

Tools like git-annex or dvc support similar strategies.

Is git-annex still alive? Last time I tried to use it, it was very rough, and the official wiki (that serves as doc + bug tracker) gives database errors trying to create an account.

Details: I wanted to have a remote I can push to but anonymous users can only pull from, couldn't piece it together.

Re: Avoid Git LFS if possible

#57

Post by ziml77 »

temac wrote: post #14

If you just don't jump on random tech without good reasons, you already naturally apply this advice. Especially since once you really need it and also wants Git, there is not much alternative (as the author recognizes). In this context, just waiting for a potential "better support for handling of large files" of official Git makes little sense; plus I make the wild prediction that what will actually happen is that it…

This is what I was thinking too. There's really nothing about Git LFS that should come as a surprise. Yes it rewrites history, but how else are you going to cut bloat from the repo after it's been stuffed in there? And the fact that the file is stored on something completely outside of git is clearly and concisely explained as the main text, directly above the download button, on https://git-lfs.github.com/

> Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.

Re: Avoid Git LFS if possible

#58

Post by klodolph »

jfim wrote: post #51
klodolph wrote: post #40

Earlier quoted context omitted.

I would say that if you care about good LFS support, that is a sufficient reason to use Mercurial. Harder to find Mercurial hosting these days, though, but I'm not worried that the Mercurial project will die off (since both Facebook and Google use it, in some manner).

Is Facebook still using mercurial? It seems that there was a blog post about it in 2014, but their repo[0] just seems to say that their codebase was originally based on/evolved from mercurial. [0] https://github.com/facebookexperimental/eden

I think this is just really the demarcation problem. At what point does it stop being “Mercurial”?

Re: Avoid Git LFS if possible

#59

Post by Dylan16807 »

tux1968 wrote: post #13
goodcjw2 wrote: post #8

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?

> $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me.

What you describe sounds fair to me. The problem is that 50GB is not a decent fraction of a hard disk.

Re: Avoid Git LFS if possible

#60

Post by Dylan16807 »

remram wrote: post #55
Someone1234 wrote: post #20

Earlier quoted context omitted.

Just for clarity, it is 50 GB of storage and 50 GB of bandwidth. So definitely not "as much as you want." If you pull it too many times you may get charged another $5.

For S3, that's $1.15 storage ($0.023/GB) + $4.50 transfer ($0.09/GB) so it's actually very comparable.

Yikes. Being comparable to amazon in price, purely because of how much amazon overcharges for bandwidth, is not exactly a flattering comparison.

If you were to add the $1.15 storage cost onto a reasonable bandwidth number like $0.01/GB you'd just about reach a third of $5.

Post reply on HN