Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

Post reply on HN
Print view View original

71–80 of 142 posts

Re: Avoid Git LFS if possible

#71

Post by petertodd »

dwohnitmok wrote: post #62
cesarb wrote: post #53

Earlier quoted context omitted.

With git-annex you have the same "one-way door" behavior: it replaces large files with a pointer to the content (in git-annex, it's a relative symbolic link which by default encodes the real file's size and hash), which is stored in git-annex's own database.

Sort of. The way that the author talks about Mercurial as not having this problem makes me think they're talking about something related but subtly different. In particular, AFAICT, Mercurial requires the exact same thing as what you're pointing out. If you want to completely disable use of largefiles then you still have to run `hg lfconvert` at some point. That also changes your revision history. The "one-way door"…

> so e.g. to move away from git-annex you can just commit the binary files directly to your git directory and then just copy them out to a separate folder whenever you go back to an old commit and re-import them.

Exactly. Here's an (anonymized) example of a git-annex symlink from one of my repos:

    ../../.git/annex/objects/AA/BB/SHA256-s123456--abcdf...1234/SHA256-s8968192--abcdf...1234
It's just a link to a file with a SHA256 hash in the name and path. The simplest way to reconstruct that in the future is to just check-in the whole `objects` directory into the repo, and copy/symlink it back to `.git/annex` when needed. You definitely don't need the git-annex software itself to view the data in the future.

I personally have hundreds of gigabytes of data in git-annex repos. It works great!

Re: Avoid Git LFS if possible

#72

Post by [deleted] »

Aeolun wrote: post #48

Did he really just try to make the argument that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? LFS has existed for several years, and as far as I know Git still doesn’t have support for large files. At this point I’m not holding out much hope.

[deleted]

Re: Avoid Git LFS if possible

#73

Post by Izkata »

remram wrote: post #56
dmm wrote: post #24

Earlier quoted context omitted.

Tools like git-annex or dvc support similar strategies.

Is git-annex still alive? Last time I tried to use it, it was very rough, and the official wiki (that serves as doc + bug tracker) gives database errors trying to create an account. Details: I wanted to have a remote I can push to but anonymous users can only pull from, couldn't piece it together.

I've always liked the simplicity of git-fat [0]:

* Initial setup includes git filter rules so that "git add" automatically uses get-fat for matching files (no need to remember to invoke git-fat when adding/changing files).

* It works by rsync'ing to/from the remote. The setup for this is in a single ".gitfat" file, separate from the filter rules.

* You do need to run "git fat push" and "git fat pull"; this can probably be automated with hooks.

So just offhand without even trying to think about the "right" way to do what you want, the committed ".gitfat" could be to a read-only remote, then you can swap it with your own un-committed file for a push that has an rsync-writeable remote.

Also, the whole thing is a single 628-line python file, so worst case it would be easy to tweak it to read something like ".gitfat-push" and not have to manually swap it.

[0] https://github.com/jedbrown/git-fat

Re: Avoid Git LFS if possible

#74

Post by rsync »

FWIW, rsync.net is currently deploying LFS support such that operations like:

  ssh user@rsync.net git clone blah
... will properly handle LFS assets, etc.

This is in response to several requests we have had for this feature...

Re: Avoid Git LFS if possible

#75

Post by neandrake »

klodolph wrote: post #58
jfim wrote: post #51

Earlier quoted context omitted.

Is Facebook still using mercurial? It seems that there was a blog post about it in 2014, but their repo[0] just seems to say that their codebase was originally based on/evolved from mercurial. [0] https://github.com/facebookexperimental/eden

I think this is just really the demarcation problem. At what point does it stop being “Mercurial”?

Losing Facebook as a contributor and community member is the larger loss, regardless of how much their internal vcs resembles or integrates with mercurial.

Re: Avoid Git LFS if possible

#76

Post by goodcjw2 »

acdha wrote: post #67
Danieru wrote: post #64

Earlier quoted context omitted.

Just be thankful you are not on Gitlab. We pay $60/year for 10GB. I need the data for my team, so we pay, but I use it as an excuse to force the team to clean up data every so often. As a game company we need large repos. It was either Gitlab or Azure DevOps.

$60/year is $5/month. You said “game” and “company” — how is that not a tiny fraction of your total costs?

Nowadays games assets are just crazily huge. AAA games tends to ship at 50G. Now think about the raw assets size and saving all the intermediate revisions.

It's quite easy to burn 1000G of storage on GitHub/GitLab (again, don't forget all the revisions). That puts just the storage cost at $6000/year. At this price point, it's really worth hosting on your own.

As Danieru mentioned they are forcing ppl to manually do cleanups, that probably indicates the storage costs are even higher which worth manual interventions.

Re: Avoid Git LFS if possible

#77

Post by Dylan16807 »

acdha wrote: post #67
Danieru wrote: post #64

Earlier quoted context omitted.

Just be thankful you are not on Gitlab. We pay $60/year for 10GB. I need the data for my team, so we pay, but I use it as an excuse to force the team to clean up data every so often. As a game company we need large repos. It was either Gitlab or Azure DevOps.

$60/year is $5/month. You said “game” and “company” — how is that not a tiny fraction of your total costs?

The problem is that it's per ten gi-- actually, looking it up, I found a page saying there is a 10GB cap!

Re: Avoid Git LFS if possible

#78

Post by neandrake »

dwohnitmok wrote: post #62
cesarb wrote: post #53

Earlier quoted context omitted.

With git-annex you have the same "one-way door" behavior: it replaces large files with a pointer to the content (in git-annex, it's a relative symbolic link which by default encodes the real file's size and hash), which is stored in git-annex's own database.

Sort of. The way that the author talks about Mercurial as not having this problem makes me think they're talking about something related but subtly different. In particular, AFAICT, Mercurial requires the exact same thing as what you're pointing out. If you want to completely disable use of largefiles then you still have to run `hg lfconvert` at some point. That also changes your revision history. The "one-way door"…

I don’t think it’s clear but mercurial has two solutions for large file support. The original “largefiles” which had all the same designs and issues as Git LFS they bring up in the blog post, and “lfs” which is newer.

I’ve used largefiles and ran into these issues and ended up having to turn it off after a few years because it’s so problematic with the tooling since it modifies the underlying mercurial commit structure like git lfs.

However it sounds like mercurial lfs is different in that it only modifies the transport layer, though I’m not totally clear on the details and have been meaning to look into it further.

Re: Avoid Git LFS if possible

#79

Post by toomanyducks »

iab wrote: post #70
wokwokwok wrote: post #52

I despise LFS. I’m sure that if you know how to use it... maybe... you can figure it out. That said; here’s my battle story: Estimate the time it’ll take to move all our repositories from a to b they said. Us: with all branches? Them: just main and develop. Us: you just clone and push to the new origin, it’s not zero but it’s trivial. Weeks later... Yeah. LFS is now banned. LFS is not a distributed version control sy…

I would rather maintain a handwritten journal of 1s and 0s than use git LFS again

honestly had no idea what git LFS was before seeing this, but okay then

Re: Avoid Git LFS if possible

#80

Post by swiley »

Someone1234 wrote: post #10

All three points are really just the same point repeated three times: That it isn't part of core/official GIT ("stop gap" until official, irreversible to later official solution, and adds complexity that an official version would lack due to extra/third party tooling). I'm frankly surprised GIT hasn't made LFS an official part by now. It fixes the problem, the problem is common and real, and GIT hasn't offered a bett…

LFS is definitely outside the scope of git.
Post reply on HN