Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Git Large File Storage

github.com

Post reply on HN
Print view View original

151–160 of 167 posts

Re: Announcing Git Large File Storage

#151

Post by bananaboy »

Impossible wrote: post #87
eropple wrote: post #64

Earlier quoted context omitted.

Most developers I know dropped Perforce for git and svn a long time ago. Configure them with the appropriate ignores and it works fine.

This hasn't been my experience working in AAA console games, although I could definitely see it being the case in mobile. What developers do you know?

Agreed, a lot of developers still use Perforce in my experience (my own indie studio included). I don't think this will make anything better for game developers generally. The big stumbling blocks are: Git is not artist/non-programmer friendly; and an inability to lock a file when editing (specifically binary files or files that are difficult to merge, e.g. complex level files like Unity's).

Re: Announcing Git Large File Storage

#152

Post by bananaboy »

jgrowl wrote: post #63
Pirate-of-SV wrote: post #24

Can't wait too see what Linus got to say about this. I suppose he got an arguably better solution to the problem?

My guess is that Linus doesn't care about large binary files.

Yeah I've read interviews where he's said that he wrote git for his use case, which I imagine doesn't include large binary files. I don't think he'd really care.

Re: Announcing Git Large File Storage

#153

Post by [deleted] »

technoweenie wrote: post #126
theli0nheart wrote: post #86

I'm sure GitHub did their due diligence before starting to work on this, but I can't lie: it bums me out a bit that they didn't find git-bigstore [1] (a project I wrote about 2 years ago) before they started, since it works in almost the exact same way. Three-line pointer files, smudge and clean filters, use of .gitattributes for which files to sync, and remote service integration. Compare "Git Large File Storage"'s…

I honestly hadn't seen git-bigstore. Git LFS started out as a Git Media update, actually. We changed the name pretty late in the process so that it didn't clash with existing Git Media repositories. Go was picked so that we could ship static binaries so its users don't have to worry about installing the correct ruby/python runtime and 3rd party dependencies. Git LFS isn't tied to any specific service either. You can…

[deleted]

Re: Announcing Git Large File Storage

#154

Post by clacke2 »

joeyh wrote: post #92
jedbrown wrote: post #88

Earlier quoted context omitted.

As the author of git-fat, I have to say the smudge/clean filter approach is a hack for large files and the performance is not good for a lot of use cases. The reality is that it's common to need fine-grained control over what files are really present in the repository, when they are cached locally, and when they are fetched over the network. Git-annex does better than the smudge/clean tools (git-fat, git-media, git-l…

Thanks for verifying my somewhat out of date guesses about smudge performance! Re the python startup time, this is particularly important for smudge/clean filters because git execs the command once per file that's being checked out (for example). I suppose even go/haskell would be a little too slow starting when checking out something like the 100k file repos some git-annex users have. ;)

Ok, that means I don't have to check out git-lfs, git-fat or git-bigstore. My annex is 250k symlinks pointing to 250 GiB of data. It's slow enough as it is.

Re: Announcing Git Large File Storage

#155

Post by willkelleher »

jedbrown wrote: post #99
SomeCallMeTim wrote: post #95

Earlier quoted context omitted.

I looked at git-fat as an option for me, but what killed it was rsync as the only backend; I really wanted to send files to S3. I also looked at git-annex, and I could see using it if it were just me on the project (or as a way of keeping fewer files on my laptop drive), but I was reluctant to add any more complexity to the source control process, since explaining how to use git-annex to the entire team was too big o…

Thanks for the feedback. There is a PR for S3 support, but it's dormant because it was mixed with other changes that broke compatibility. I haven't personally wanted S3, so haven't made time to rework the PR.

Ah, that is my fault. We're using the fork quite actively, but need to revive that PR and improve the config settings.

Re: Announcing Git Large File Storage

#156

Post by eropple »

Impossible wrote: post #87
eropple wrote: post #64

Earlier quoted context omitted.

Most developers I know dropped Perforce for git and svn a long time ago. Configure them with the appropriate ignores and it works fine.

This hasn't been my experience working in AAA console games, although I could definitely see it being the case in mobile. What developers do you know?

Mobile and mid-tier console developers. Not AAA, but with budgets firmly in the "you spent what, for that?" range.

Re: Announcing Git Large File Storage

#158

Post by dhruvgupta »

Tiktaalik wrote: post #128
duartetb wrote: post #43

Does this mean gamedevs might start droping Perforce for this? If its not too expensive maybe?

This is certainly the issue that is preventing game devs from adopting Git. On the other hand game devs at this point are very used to Perforce, and it looks like Perforce is interested in solving this problem from the other side, by adding Git features to Perforce Helix and making it distributed.

Helix Versioning Engine is a native DVCS. This is in addition to a Git management solution that is part of the solution. Choice of workflow, combined with efficiency in large file, large repo handling -- definitely interested in solving the problem right, instead of using band-aid.

Re: Announcing Git Large File Storage

#159

Post by paulcody »

joeyh wrote: post #89
jefurii wrote: post #30

Earlier quoted context omitted.

This and git-annex (and git-fat and others) use the same basic architecture of storing links in Git and schlepping the binaries around separately. Git-annex renames binaries with their SHA256 hashes, puts them in a .git/annex/ dir, and replaces files in the working dir with symlinks. Git-LFS seems to use small metadata pointer files (SHA256 hash, file size, git-lfs version) instead of symlinks. Not sure whether the f…

The lack of location tracking looks like the most significant difference to me. While the git-lfs documentation does mention that different git remotes can have different LFS endpoints configured, all git-lfs knows about a file is its SHA256. So how can it tell which remote to download the file from? The best it could do is try different remotes until it finds one that has the file. I hesitate to say this means git-l…

At a minimum, the metafile should include a URI, stating the last known location of the file, rather than just a bare SHA256.

Re: Announcing Git Large File Storage

#160

Post by joeyh »

clacke2 wrote: post #154
joeyh wrote: post #92

Earlier quoted context omitted.

Thanks for verifying my somewhat out of date guesses about smudge performance! Re the python startup time, this is particularly important for smudge/clean filters because git execs the command once per file that's being checked out (for example). I suppose even go/haskell would be a little too slow starting when checking out something like the 100k file repos some git-annex users have. ;)

Ok, that means I don't have to check out git-lfs, git-fat or git-bigstore. My annex is 250k symlinks pointing to 250 GiB of data. It's slow enough as it is.

At 250k files in one branch, you are starting to run into other scalability limits in git too, like the innefficient method it uses to update .git/index (rewriting the whole thing).
Post reply on HN