Live data from Hacker News

Announcing Git Large File Storage

github.com

31–40 of 167 posts

Re: Announcing Git Large File Storage

#31
post #16

So basically it's git-annex, but tied to GitHub. http://git-annex.branchable.com/

> tied to GitHub.

The protocol is open (https://github.com/github/git-lfs/blob/master/docs/api.md) and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server.

edit: added protocol spec

Re: Announcing Git Large File Storage

#32

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeable binary assets. Otherwise you get two people touching the same file and someone has to destroy their changes. That never makes anyone happy. It's also unseen how good the disk performance is. These two areas are the reason why Perforce is still my go-to solut…

While the use case you describe isn't solved by this, other use cases are. For example, wanting to easily access versioned, compiled binaries for a revision.

Re: Announcing Git Large File Storage

#33

Earlier quoted context omitted.

I would suspect the point is rather that you have a bunch of megabyte range files, and you rarely update them and don't have to sync. But for most workflows this feature seems targeted at, the free tier seems insufficient.

I'm having trouble seeing where a 1GB/month quota in any way meshes with "large file" support. The free tier is basically "test out the API, don't even think about using it for real".

Yes, that is the free tier. If you want to use it seriously, it will cost some money. OR you can use it and host your own file server, for free.

I don't think these facts are a problem. They create an open source tool, provide a location to try it out, and a service to pay to use it if you like it and don't want to host yourself. Seems like a fair offer.

Re: Announcing Git Large File Storage

#34
post #27

Earlier quoted context omitted.

That's an interesting point I hadn't considered before. Git, as a distributed vcs, takes the position that everyone can edit all files, and rare conflicts can be managed easily because changes are relatively small and diffable. But these assumptions break down with binary assets. Changes are not small, they typically change entire files at once. They're also not diffable. As a result, conflicts are not rare, they're…

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.

Re: Announcing Git Large File Storage

#35
post #16

So basically it's git-annex, but tied to GitHub. http://git-annex.branchable.com/

(Not a git-annex user here). I suppose functionally, these two are similar. But the use case is different. git-annex seems to be more for managing files and making sure they don't disappear on you. GitHub's new thing is for keeping track of larger objects inside your git project efficiently. Basically, yeah, you can use git-annex to store the PSD, the audio samples, the promo video, etc. but wouldn't it be nice to ha…

> (Not a git-annex user here)

You could at least read the examples on the git-annex page[1] before passing judgement that the use cases are at all different (they're not). Instead of using a new 'lfs' command that ties you to GitHub, you use an 'annex' command (along with a few others).

Git-annex does just fine "keeping track of larger objects inside your git project efficiently", and is no more divorced from your normal project workflow than GitHub's lfs.

[1]http://git-annex.branchable.com/git-annex/

Re: Announcing Git Large File Storage

#36

Earlier quoted context omitted.

I have a feeling the decision for how to arrange, as a separate thing, is likely to feed the monetization component. Particularly if it's limited to using GitHub's storage.

Specs are open, there's an open server implementation. It might be easiest to set it up with github, but if it catches on, I expect implementations will be readily available from all github-like platforms and as stand-alone.

Awesome. I did notice they called it "Git LFS" instead of "GitHub LFS", which should be a clue there, though from other comments I figured it might be GitHub specific.

Re: Announcing Git Large File Storage

#37

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeable binary assets. Otherwise you get two people touching the same file and someone has to destroy their changes. That never makes anyone happy. It's also unseen how good the disk performance is. These two areas are the reason why Perforce is still my go-to solut…

That's an interesting point I hadn't considered before. Git, as a distributed vcs, takes the position that everyone can edit all files, and rare conflicts can be managed easily because changes are relatively small and diffable. But these assumptions break down with binary assets. Changes are not small, they typically change entire files at once. They're also not diffable. As a result, conflicts are not rare, they're…

Note that GitHub itself does create useful diffs of images:

https://github.com/cameronmcefee/Image-Diff-View-Modes/commi...

More: https://github.com/blog/817-behold-image-view-modes

Re: Announcing Git Large File Storage

#38
post #27

Earlier quoted context omitted.

That's an interesting point I hadn't considered before. Git, as a distributed vcs, takes the position that everyone can edit all files, and rare conflicts can be managed easily because changes are relatively small and diffable. But these assumptions break down with binary assets. Changes are not small, they typically change entire files at once. They're also not diffable. As a result, conflicts are not rare, they're…

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge.

How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations in a format like png.

How do you diff changes? If we have a raster image and one person changes one thing by a small amount, say increases brightness 1%, this could alter every pixel of the image! How would you detect that change and interleave it with something like a contrast adjustment of 1% that could also change every pixel? Sure, the merger would still have to choose which adjustment goes first if the changes aren't independent, but how would they know that's what changed? I.e. how would the diff tool know that the changes are "brightness +1%" and "contrast +1%" and not some other arbitrary number of adjustments?

Re: Announcing Git Large File Storage

#39
post #30
post #18

Has someone had a closer look and can say how this compares to Git-Annex?

This and git-annex (and git-fat and others) use the same basic architecture of storing links in Git and schlepping the binaries around separately. Git-annex renames binaries with their SHA256 hashes, puts them in a .git/annex/ dir, and replaces files in the working dir with symlinks. Git-LFS seems to use small metadata pointer files (SHA256 hash, file size, git-lfs version) instead of symlinks. Not sure whether the f…

git annex uses symlinks in indirect mode, but can use small files in direct mode (useful for filesystems which don't support symlinks, like many Android sdcards).

Re: Announcing Git Large File Storage

#40
post #31
post #16

So basically it's git-annex, but tied to GitHub. http://git-annex.branchable.com/

> tied to GitHub. The protocol is open ( https://github.com/github/git-lfs/blob/master/docs/api.md ) and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server . edit: added protocol spec

> The protocol is open and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server.

This isn't about this particular instance (Github's LFS), but in general, a "reference implementation" isn't the same thing as having an open protocol.

Having a reference implementation without a proper specification means that any other implementations have to re-implement the existing reference implementation, including any bugs. The purpose of a specification is to outline undefined behavior as much as it is to outline defined behavior. That is, the specification says, "these are the portions of the program which you may not rely on".

We've seen this happen in some languages in which a particular implementation is either the de facto or de jure standard. Other compilers or interpreters end up having to mimic their bugs when it comes to things like arithmetic overflow/precision errors, because developers have come to rely on the language behaving one way, in the absence of any clear rules telling them otherwise[0].

[0] Not that developers may not rely on things that a specification explicitly tells them not to - there are plenty of examples of this too - but at least then it's possible to say determine either that a particular program will run on any standards-compliant implementation, or that it is implementation-specific.

Post reply on HN