Announcing Git Large File Storage

#1

Post by dewski » Wed, Apr 08, 2015, 7:02 PM UTC

Announcing Git Large File Storage
github.com

Re: Announcing Git Large File Storage

#2

Post by m0th87 » Wed, Apr 08, 2015, 7:15 PM UTC

Our solution is likely a lot more duct tape-y, but we developed a straight-forward tool in Go for managing large assets in git: https://github.com/dailymuse/git-fit

There's a number of other solutions open source out there, some of which are documented in our readme.

Re: Announcing Git Large File Storage

#3

Post by justinsb » Wed, Apr 08, 2015, 7:18 PM UTC

This solves a real problem, but I can't help but feel it is a band-aid hack.

The main fundamental advantage (vs implementation quirks of git) I can see is that these files are only fetched on a git checkout. But (of course) this breaks offline support, and it requires additional user action.

Wouldn't it have been fairly easy to build exactly the same functionality into git itself? "Big" blobs aren't fetched until they are checked-out? This also has the advantage the definition of "big" could depend on your connectivity / disk space / whatever, rather than being set per-repo.

Re: Announcing Git Large File Storage

#4

Post by jxf » Wed, Apr 08, 2015, 7:19 PM UTC

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server.

What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only supports their own storage endpoint. So you couldn't use, say, S3 to host your Git LFS files.

Re: Announcing Git Large File Storage

#5

Post by geoffreyirving » Wed, Apr 08, 2015, 7:20 PM UTC

After a quick scan, I'm a bit worried that this is too tied to a server in practice. For example, if I've downloaded everything locally, can I easily clone the whole download (including all lfs files) into a separate repo? If I can, can changes to each be swapped back and forth?

Re: Announcing Git Large File Storage

#6

Post by ocdtrekkie » Wed, Apr 08, 2015, 7:21 PM UTC

justinsb wrote: post #3

This solves a real problem, but I can't help but feel it is a band-aid hack. The main fundamental advantage (vs implementation quirks of git) I can see is that these files are only fetched on a git checkout. But (of course) this breaks offline support, and it requires additional user action. Wouldn't it have been fairly easy to build exactly the same functionality into git itself? "Big" blobs aren't fetched until the…

I have a feeling the decision for how to arrange, as a separate thing, is likely to feed the monetization component. Particularly if it's limited to using GitHub's storage.

Re: Announcing Git Large File Storage

#7

Post by icebraining » Wed, Apr 08, 2015, 7:21 PM UTC

jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server.

That's exactly what git-annex does. Except it can host on your own servers, or S3, or Tahoe-LAFS, or rsync.net, etc. And it's free software. And it supports multiple servers for the same repo, so you have redundancy.

Adding an S3 remote is just setting the AWS keys and running a single command: http://git-annex.branchable.com/tips/using_Amazon_S3/

Re: Announcing Git Large File Storage

#8

Post by cbhl » Wed, Apr 08, 2015, 7:21 PM UTC

jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

If the extension is open-source, you could probably build your own server implementation and fork the extension to change what server it contacts.

Re: Announcing Git Large File Storage

#9

Post by [deleted] » Wed, Apr 08, 2015, 7:22 PM UTC

jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

[deleted]

Re: Announcing Git Large File Storage

#10

Post by Oompa » Wed, Apr 08, 2015, 7:22 PM UTC

jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

There's an example server that was also open sourced today: https://github.com/github/lfs-test-server