Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Git Large File Storage

github.com

Post reply on HN
Print view View original

121–130 of 167 posts

Re: Announcing Git Large File Storage

#121

Post by tclem »

theli0nheart wrote: post #86

I'm sure GitHub did their due diligence before starting to work on this, but I can't lie: it bums me out a bit that they didn't find git-bigstore [1] (a project I wrote about 2 years ago) before they started, since it works in almost the exact same way. Three-line pointer files, smudge and clean filters, use of .gitattributes for which files to sync, and remote service integration. Compare "Git Large File Storage"'s…

A lot of inspiration did come from the git-media project which goes back to at least 2009: https://github.com/alebedev/git-media/commit/705ea59bd98a3d1...

git-bigstore is awesome too though!

Re: Announcing Git Large File Storage

#122

Post by sytse »

theli0nheart wrote: post #86

I'm sure GitHub did their due diligence before starting to work on this, but I can't lie: it bums me out a bit that they didn't find git-bigstore [1] (a project I wrote about 2 years ago) before they started, since it works in almost the exact same way. Three-line pointer files, smudge and clean filters, use of .gitattributes for which files to sync, and remote service integration. Compare "Git Large File Storage"'s…

It seems to be based on git-media initially, would be interesting to know why they changed that https://github.com/github/git-lfs/commit/10a8eceefdb081edf61...

Re: Announcing Git Large File Storage

#124

Post by sytse »

sytse wrote: post #123

To celebrate the broader support for git with large files we just raised the storage limit of GitLab.com to 10GB https://about.gitlab.com/2015/04/08/gitlab-dot-com-storage-l... also, we're glad GitHub open sourced it and didn't call it assman

Someone asked if this was temporarily or permanent, it is permanent, see https://news.ycombinator.com/item?id=9344984

Re: Announcing Git Large File Storage

#125

Post by Rapzid »

sytse wrote: post #68
efuquen wrote: post #57

Earlier quoted context omitted.

Why reinvent the same thing? What were the technical deficiencies of git-annex that necessitated this? Or is this NIH syndrome? I think these are all reasonable questions.

GitLab CEO here. It would have been nice if they would have developed this in the open, Joey from git-annex is pretty open minded, maybe we could have prevented another standard.

I would have thought that it would have been particularly important for this to have been open to review and critique from the start given GitHub's status and importance to the wider community. Heck, even Microsoft is developing new .Net components in the open.

Would have been nice to have had debate on existing solutions to weigh the pros/cons.

Re: Announcing Git Large File Storage

#126

Post by technoweenie »

theli0nheart wrote: post #86

I'm sure GitHub did their due diligence before starting to work on this, but I can't lie: it bums me out a bit that they didn't find git-bigstore [1] (a project I wrote about 2 years ago) before they started, since it works in almost the exact same way. Three-line pointer files, smudge and clean filters, use of .gitattributes for which files to sync, and remote service integration. Compare "Git Large File Storage"'s…

I honestly hadn't seen git-bigstore. Git LFS started out as a Git Media update, actually. We changed the name pretty late in the process so that it didn't clash with existing Git Media repositories. Go was picked so that we could ship static binaries so its users don't have to worry about installing the correct ruby/python runtime and 3rd party dependencies.

Git LFS isn't tied to any specific service either. You can install our reference server somewhere, and start using it with your GitHub (or any host really) repositories without having to sit in our wait list or pay us a dime. Though our reference server isn't really production ready, so I wouldn't advise that for real work just yet :)

Re: Announcing Git Large File Storage

#127

Post by sytse »

Rapzid wrote: post #125
sytse wrote: post #68

Earlier quoted context omitted.

GitLab CEO here. It would have been nice if they would have developed this in the open, Joey from git-annex is pretty open minded, maybe we could have prevented another standard.

I would have thought that it would have been particularly important for this to have been open to review and critique from the start given GitHub's status and importance to the wider community. Heck, even Microsoft is developing new .Net components in the open. Would have been nice to have had debate on existing solutions to weigh the pros/cons.

Thanks Rapzid, I also think that an open process would have lead to a better outcome. I hope GitHub follows up with a rationale for the points Joey mentioned in this thread.

Re: Announcing Git Large File Storage

#128

Post by Tiktaalik »

duartetb wrote: post #43

Does this mean gamedevs might start droping Perforce for this? If its not too expensive maybe?

This is certainly the issue that is preventing game devs from adopting Git.

On the other hand game devs at this point are very used to Perforce, and it looks like Perforce is interested in solving this problem from the other side, by adding Git features to Perforce Helix and making it distributed.

Re: Announcing Git Large File Storage

#129

Post by caust1c »

joeyh wrote: post #92
jedbrown wrote: post #88

Earlier quoted context omitted.

As the author of git-fat, I have to say the smudge/clean filter approach is a hack for large files and the performance is not good for a lot of use cases. The reality is that it's common to need fine-grained control over what files are really present in the repository, when they are cached locally, and when they are fetched over the network. Git-annex does better than the smudge/clean tools (git-fat, git-media, git-l…

Thanks for verifying my somewhat out of date guesses about smudge performance! Re the python startup time, this is particularly important for smudge/clean filters because git execs the command once per file that's being checked out (for example). I suppose even go/haskell would be a little too slow starting when checking out something like the 100k file repos some git-annex users have. ;)

Yep. It's really a problem that needs to be fixed in git proper. I'm surpised that github of all people didn't realize this and/or invest the time to do it right.

The one major obvious drawback for it being fixed in git is that it's not backwards compatible with old clients though. Doing it in go is probably a good improvement over the existing solutions of git-media and git-fat but I don't think is the final one.

Funny enough, although I had thought this since I started working with git-fat, I only recently admitted it[1]. Perhaps if I had admitted it when I first started work on it then there's a chance they would have seen it! :-P

[1]https://github.com/cyaninc/git-fat/issues/41#issuecomment-88...

Re: Announcing Git Large File Storage

#130

Post by caust1c »

mdonahoe wrote: post #115
jedbrown wrote: post #88

Earlier quoted context omitted.

As the author of git-fat, I have to say the smudge/clean filter approach is a hack for large files and the performance is not good for a lot of use cases. The reality is that it's common to need fine-grained control over what files are really present in the repository, when they are cached locally, and when they are fetched over the network. Git-annex does better than the smudge/clean tools (git-fat, git-media, git-l…

So if I have large (1GB+) files in my repo, you recommend against git-fat? I have been enjoying the simplicity of git-fat, but running git diff and especially git grep makes me think I should switch to something else.

As someone who worked on it a lot, I'd say it'd be worth it to switch to git-lfs. Exactly the same designs with different formats but written in golang.
Post reply on HN