Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Git Large File Storage

github.com

Post reply on HN
Print view View original

111–120 of 167 posts

Re: Announcing Git Large File Storage

#111

Post by [deleted] »

joeyh wrote: post #89
jefurii wrote: post #30

Earlier quoted context omitted.

This and git-annex (and git-fat and others) use the same basic architecture of storing links in Git and schlepping the binaries around separately. Git-annex renames binaries with their SHA256 hashes, puts them in a .git/annex/ dir, and replaces files in the working dir with symlinks. Git-LFS seems to use small metadata pointer files (SHA256 hash, file size, git-lfs version) instead of symlinks. Not sure whether the f…

The lack of location tracking looks like the most significant difference to me. While the git-lfs documentation does mention that different git remotes can have different LFS endpoints configured, all git-lfs knows about a file is its SHA256. So how can it tell which remote to download the file from? The best it could do is try different remotes until it finds one that has the file. I hesitate to say this means git-l…

[deleted]

Re: Announcing Git Large File Storage

#112

Post by sytse »

spullara wrote: post #104
theli0nheart wrote: post #86

I'm sure GitHub did their due diligence before starting to work on this, but I can't lie: it bums me out a bit that they didn't find git-bigstore [1] (a project I wrote about 2 years ago) before they started, since it works in almost the exact same way. Three-line pointer files, smudge and clean filters, use of .gitattributes for which files to sync, and remote service integration. Compare "Git Large File Storage"'s…

Very cool. Maybe this will shine a light on your project. First thing I looked for in their announcement was S3 support...

If you want s3 support consider using git-annex. GitLab.com and GitLab Enterprise Edition support it out of the box.

Re: Announcing Git Large File Storage

#113

Post by Animats »

Does Github really do this using git's "smudge" and "clean" filters? That would mean reprocessing the whole file for each access. That's inefficient. It's useful only if someone else is paying for the disk bandwidth, and necessary only if you don't have control of the storage system. Why would GitHub do that to itself?

Re: Announcing Git Large File Storage

#114

Post by icebraining »

sytse wrote: post #110
icebraining wrote: post #26

Earlier quoted context omitted.

The filter-by-filetype can be replace by small script that augments git: http://git-annex.branchable.com/forum/help_running_git-annex...

Would be nice to replace that with git hooks so you can just use regular git commands. Any idea's if that is feasible?

I'm not sure, but I doubt a hook would do. An alternative would be to have a frontend script to git that would shadow the git command (using shell aliases) and call git-annex when appropriate.

Re: Announcing Git Large File Storage

#115

Post by mdonahoe »

jedbrown wrote: post #88
joeyh wrote: post #72

It's interesting that this uses smudge/clean filters. When I considered using those for git-annex, I noticed that the smudge and clean filters both had to consume the entire content of the file from stdin. Which means that eg, git status will need to feed all the large files in your work tree into git-lfs's smudge filter. I'm interested to see how this scales. My feeling when I looked at it was that it was not suffic…

As the author of git-fat, I have to say the smudge/clean filter approach is a hack for large files and the performance is not good for a lot of use cases. The reality is that it's common to need fine-grained control over what files are really present in the repository, when they are cached locally, and when they are fetched over the network. Git-annex does better than the smudge/clean tools (git-fat, git-media, git-l…

So if I have large (1GB+) files in my repo, you recommend against git-fat?

I have been enjoying the simplicity of git-fat, but running git diff and especially git grep makes me think I should switch to something else.

Re: Announcing Git Large File Storage

#117

Post by thoughtflretort »

sytse wrote: post #103
FireBeyond wrote: post #85

Earlier quoted context omitted.

Because they are going to charge for LFS, I suspect. Just as they don't support private repos for free, either.

They could also charge for git-annex I think. It is not that they need their own type of private repo's to charge for that. By the way, GitLab.com offers unlimited repo's and 5GB per repo for free.

How many users/how long will you be able to provide that for free?

Re: Announcing Git Large File Storage

#118

Post by tomphoolery »

jedbrown wrote: post #96
theli0nheart wrote: post #86

I'm sure GitHub did their due diligence before starting to work on this, but I can't lie: it bums me out a bit that they didn't find git-bigstore [1] (a project I wrote about 2 years ago) before they started, since it works in almost the exact same way. Three-line pointer files, smudge and clean filters, use of .gitattributes for which files to sync, and remote service integration. Compare "Git Large File Storage"'s…

Looks like you wrote git-bigstore a few months after I wrote git-fat (also Python and a similar design; partially inspired by git-media). It would be interesting to do some performance comparisons and merge our capabilities, perhaps with support for each other's stub formats if we can do it in a compatible way.

It's also somewhat notable to mention that git-media was written by one of the founders of GitHub. So...that may be why? :)

Re: Announcing Git Large File Storage

#119

Post by sytse »

thoughtflretort wrote: post #117
sytse wrote: post #103

Earlier quoted context omitted.

They could also charge for git-annex I think. It is not that they need their own type of private repo's to charge for that. By the way, GitLab.com offers unlimited repo's and 5GB per repo for free.

How many users/how long will you be able to provide that for free?

Forever, our business model is to make money with on-premises software (GitLab EE). In the long term we might add a marketplace to GitLab.com were you can subscribe to additional services (like Heroku does).

Re: Announcing Git Large File Storage

#120

Post by sytse »

icebraining wrote: post #114
sytse wrote: post #110

Earlier quoted context omitted.

Would be nice to replace that with git hooks so you can just use regular git commands. Any idea's if that is feasible?

I'm not sure, but I doubt a hook would do. An alternative would be to have a frontend script to git that would shadow the git command (using shell aliases) and call git-annex when appropriate.

Yeah, I don't like shell aliases but it would work. I wonder how the LFS client works.
Post reply on HN