Announcing Git Large File Storage • Hacker News Board

Re: Announcing Git Large File Storage

#51

Post by res0nat0r » Wed, Apr 08, 2015, 8:33 PM UTC

fragmede wrote: post #35

IgorPartola wrote: post #29

Earlier quoted context omitted.

(Not a git-annex user here). I suppose functionally, these two are similar. But the use case is different. git-annex seems to be more for managing files and making sure they don't disappear on you. GitHub's new thing is for keeping track of larger objects inside your git project efficiently. Basically, yeah, you can use git-annex to store the PSD, the audio samples, the promo video, etc. but wouldn't it be nice to ha…

> (Not a git-annex user here) You could at least read the examples on the git-annex page[1] before passing judgement that the use cases are at all different (they're not). Instead of using a new 'lfs' command that ties you to GitHub, you use an 'annex' command (along with a few others). Git-annex does just fine "keeping track of larger objects inside your git project efficiently", and is no more divorced from your no…

I tried git-annex a couple of times to sync my 2 OSX and linux based computers at my house and play around. It wasn't the easiest thing to get setup and working and I couldn't get one of the OSX hosts to work at all.

I'm sure this will be much easier to use for the end user like other github products and will "just work" out of the box.

Re: Announcing Git Large File Storage

#52

Post by res0nat0r » Wed, Apr 08, 2015, 8:35 PM UTC

vvanders wrote: post #14

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeable binary assets. Otherwise you get two people touching the same file and someone has to destroy their changes. That never makes anyone happy. It's also unseen how good the disk performance is. These two areas are the reason why Perforce is still my go-to solut…

This looks to be competing with AWS CodeCommit too which will allow any size of files to be committed since it is all just backed by S3.

http://aws.amazon.com/codecommit/

Re: Announcing Git Large File Storage

#53

Post by lesplat » Wed, Apr 08, 2015, 8:35 PM UTC

So does this mean the large files are actually versioned?

Re: Announcing Git Large File Storage

#54

Post by jbramble » Wed, Apr 08, 2015, 8:39 PM UTC

Does this mean github could become useful for music production?

Re: Announcing Git Large File Storage

#55

Post by rmc » Wed, Apr 08, 2015, 8:44 PM UTC

bhuga wrote: post #31

Doji wrote: post #16

So basically it's git-annex, but tied to GitHub. http://git-annex.branchable.com/

> tied to GitHub. The protocol is open ( https://github.com/github/git-lfs/blob/master/docs/api.md ) and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server . edit: added protocol spec

Embrace. Extend. Extinguish.

Re: Announcing Git Large File Storage

#56

Post by efuquen » Wed, Apr 08, 2015, 8:44 PM UTC

azernik wrote: post #41

icebraining wrote: post #7

Earlier quoted context omitted.

You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly what git-annex does. Except it can host on your own servers, or S3, or Tahoe-LAFS, or rsync.net, etc. And it's free software. And it supports multiple servers for the same repo, so you have redundancy. Adding an S3 remote is just s…

This is also free software, and you can also use your own server.

I think the question is why did they role their own solution when there was one already an open and freely one available. If it wasn't suitable in some way I would really like to know why.

Re: Announcing Git Large File Storage

#57

Post by efuquen » Wed, Apr 08, 2015, 8:46 PM UTC

bhuga wrote: post #31

Doji wrote: post #16

So basically it's git-annex, but tied to GitHub. http://git-annex.branchable.com/

> tied to GitHub. The protocol is open ( https://github.com/github/git-lfs/blob/master/docs/api.md ) and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server . edit: added protocol spec

Why reinvent the same thing? What were the technical deficiencies of git-annex that necessitated this? Or is this NIH syndrome? I think these are all reasonable questions.

Re: Announcing Git Large File Storage

#58

Post by lebek » Wed, Apr 08, 2015, 8:53 PM UTC

jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

Nitpick: they gave four examples of large binary files: audio samples, datasets, graphics, and videos. Diffs would be meaningful in every case.

Re: Announcing Git Large File Storage

#59

Post by Oompa » Wed, Apr 08, 2015, 8:55 PM UTC

infogulch wrote: post #38

jordigh wrote: post #27

Earlier quoted context omitted.

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations i…

Showing diffs for binary assets doesn't need to include things like "brightness 1%". GitHub currently supports image diffs, they're simply displayed side by side, or on top of each other.

Re: Announcing Git Large File Storage

#60

Post by ansible » Wed, Apr 08, 2015, 9:06 PM UTC

Rondom wrote: post #18

Has someone had a closer look and can say how this compares to Git-Annex?

I think it is much closer to git-fat [1] (as jefurii mentions).

I've been trying out git-fat on a large repository that has some binaries in it. Since we are already using ssh to access the server (and have the usual ssh-agent setup), it was easy to integrate it with scp.

Unfortunately, the performance for our workload (45K files, 5GB total) with git-fat wasn't much different than plain git.

It seems most of my problems stem from the number of files, rather than the size. If I had a smaller number of very large files, git-fat might be a good solution.

[1] https://github.com/jedbrown/git-fat