Live data from Hacker News

Announcing Git Large File Storage

github.com

71–80 of 167 posts

Re: Announcing Git Large File Storage

#71
post #55
post #31

Earlier quoted context omitted.

> tied to GitHub. The protocol is open ( https://github.com/github/git-lfs/blob/master/docs/api.md ) and the client additions are open source. There is a reference server implementation at https://github.com/github/lfs-test-server . edit: added protocol spec

Embrace. Extend. Extinguish.

All GitLab contributors are working on an alternative ending.

Re: Announcing Git Large File Storage

#72
It's interesting that this uses smudge/clean filters. When I considered using those for git-annex, I noticed that the smudge and clean filters both had to consume the entire content of the file from stdin. Which means that eg, git status will need to feed all the large files in your work tree into git-lfs's smudge filter.

I'm interested to see how this scales. My feeling when I looked at it was that it was not sufficiently scalable without improving the smudge/clean filter interface. I mentioned this to the git devs at the time and even tried to develop a patch, but AFAICS, nothing yet.

Details: " rel="nofollow">https://git-annex.branchable.com/todo/smudge>

Re: Announcing Git Large File Storage

#73
post #48

Earlier quoted context omitted.

A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.

> A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would le…

What about compressing the same image with two different encoders? Or with different zlib settings? Even if the diff after lossless decompression is small, you have to somehow reconstruct the decisions that the compressor made on the target file.

Re: Announcing Git Large File Storage

#74
post #48

Earlier quoted context omitted.

A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.

> A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would le…

[deleted]

Re: Announcing Git Large File Storage

#75
Is there any hint on pricing? Slighty annoying to have a section titled "Pricing" which .... doesn't tell you the price. I would much rather use my own external server for hosting large files, it is going to need to be price competitive with other options to be interesting I would think.

Re: Announcing Git Large File Storage

#76
post #72

It's interesting that this uses smudge/clean filters. When I considered using those for git-annex, I noticed that the smudge and clean filters both had to consume the entire content of the file from stdin. Which means that eg, git status will need to feed all the large files in your work tree into git-lfs's smudge filter. I'm interested to see how this scales. My feeling when I looked at it was that it was not suffic…

Seems that git status nowadays does manage to avoid running the smudge filter, unless the file's stat has changed. This overhead does still exist for other operations, like git checkout.

Re: Announcing Git Large File Storage

#77
post #44

Earlier quoted context omitted.

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations i…

I don't think the problem is trivial, but I don't think it's hopeless either. If we can measure a person's pulse and mood with a camera pointed at their face, I'm sure we can come up with a tool that can approximate semantically meaningful diffs of artwork. For image formats like xcf that store parts of the image or the editing history independently, this problem becomes even more tractable.

It's very tractable if you don't insist on the diff reconstructing the target file byte-for-byte. But this would require pretty invasive changes in the version control system, no?

If you change the top-left pixel of a PNG, for example, between the intra prediction and the DEFLATE compression, the new file can be totally different, and to reconstruct it you either hope the destination is using the exact same libpng with the exact same settings, or you have to find a space-efficient way to write down all the arbitrary encoding decisions the format allows.

Re: Announcing Git Large File Storage

#79
post #72

It's interesting that this uses smudge/clean filters. When I considered using those for git-annex, I noticed that the smudge and clean filters both had to consume the entire content of the file from stdin. Which means that eg, git status will need to feed all the large files in your work tree into git-lfs's smudge filter. I'm interested to see how this scales. My feeling when I looked at it was that it was not suffic…

Also, is there any reason Git LFS can't be used as a special remote for git-annex?

It would provide an easy way for people to host their git-annex repos entirely on GitHub.

Re: Announcing Git Large File Storage

#80

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeable binary assets. Otherwise you get two people touching the same file and someone has to destroy their changes. That never makes anyone happy. It's also unseen how good the disk performance is. These two areas are the reason why Perforce is still my go-to solut…

Anyone who currently stores large files in git already has this problem; git-lfs is strictly better than plain old git in this case.

Format-specific diffing and merging tools that are aware of git-lfs would probably help, and those can come later.

Post reply on HN