Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Git Large File Storage

github.com

Post reply on HN
Print view View original

41–50 of 167 posts

Re: Announcing Git Large File Storage

#41

Post by azernik »

icebraining wrote: post #7
jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly what git-annex does. Except it can host on your own servers, or S3, or Tahoe-LAFS, or rsync.net, etc. And it's free software. And it supports multiple servers for the same repo, so you have redundancy. Adding an S3 remote is just s…

This is also free software, and you can also use your own server.

Re: Announcing Git Large File Storage

#44

Post by jordigh »

infogulch wrote: post #38
jordigh wrote: post #27

Earlier quoted context omitted.

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations i…

I don't think the problem is trivial, but I don't think it's hopeless either. If we can measure a person's pulse and mood with a camera pointed at their face, I'm sure we can come up with a tool that can approximate semantically meaningful diffs of artwork. For image formats like xcf that store parts of the image or the editing history independently, this problem becomes even more tractable.

Re: Announcing Git Large File Storage

#45

Post by bhuga »

chimeracoder wrote: post #40
bhuga wrote: post #31

Earlier quoted context omitted.

> tied to GitHub. The protocol is open ( https://github.com/github/git-lfs/blob/master/docs/api.md ) and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server . edit: added protocol spec

> The protocol is open and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server . This isn't about this particular instance (Github's LFS), but in general, a "reference implementation" isn't the same thing as having an open protocol. Having a reference implementation without a proper specification means that any other implementations have to re-…

You're right, the protocol is also required. I didn't link to it in my comment, but it is also open and well-defined. I've updated my comment. Thanks!

Re: Announcing Git Large File Storage

#46

Post by jordigh »

jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

> You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server.

That's exactly how Mercurial's largefiles works too:

http://mercurial.selenic.com/wiki/LargefilesExtension#The_lo...

Also, you don't need any kind of special server. Any hg repo can turn into a largefiles store by just flipping the bit in the repo configuration.

Re: Announcing Git Large File Storage

#47

Post by rakoo »

icebraining wrote: post #7
jxf wrote: post #4

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only…

You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly what git-annex does. Except it can host on your own servers, or S3, or Tahoe-LAFS, or rsync.net, etc. And it's free software. And it supports multiple servers for the same repo, so you have redundancy. Adding an S3 remote is just s…

And if you want the simplest solution (ie store blobs with the rest of your code), gitlab offers git-annex compatibility (https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-...)

Re: Announcing Git Large File Storage

#48

Post by Thrymr »

WallWextra wrote: post #34
jordigh wrote: post #27

Earlier quoted context omitted.

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.

> A bit-exact differ for, say, jpegs, probably wouldn't work all that well.

Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would lead to more complex diffs.

Re: Announcing Git Large File Storage

#50

Post by rakoo »

infogulch wrote: post #38
jordigh wrote: post #27

Earlier quoted context omitted.

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations i…

I don't think those are your scm's business. git is the stupid content tracker, it tracks whatever you push into it.

If I were to make a contrived analogy, how do you know how to diff random arrays of bytes ? Where do you start, where do you stop ? How do you know that "\n" or "\r\n" is some kind of delimiter ? You put that knowledge in "diff" and in your editor, and git stores the raw array of bytes. It's the same with binary content: git doesn't care that you don't deal with UTF-8 characters, it doesn't care that it isn't bounded by newline characters.

If you take things this way, you start to understand that the "diff" tool you use must be appropriate to the content you have, and it's not the scm's business. Now, how exactly would a diff work for images, I have absolutely no idea.

Post reply on HN