Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Git Large File Storage

github.com

Post reply on HN
Print view View original

131–140 of 167 posts

Re: Announcing Git Large File Storage

#131

Post by caust1c »

sytse wrote: post #68
efuquen wrote: post #57

Earlier quoted context omitted.

Why reinvent the same thing? What were the technical deficiencies of git-annex that necessitated this? Or is this NIH syndrome? I think these are all reasonable questions.

GitLab CEO here. It would have been nice if they would have developed this in the open, Joey from git-annex is pretty open minded, maybe we could have prevented another standard.

To be honest, it doesn't look like they put much additional thought into the problem at all. git-filters are going to be the limiting factor for performance here and depending on your filesize might still cause a considerable slowdown. I would have liked to see a solution with patching git and `sendfile` myself.

See my other comment here: https://news.ycombinator.com/item?id=9345242

Re: Announcing Git Large File Storage

#132

Post by greggman »

mjs wrote: post #37
infogulch wrote: post #25

Earlier quoted context omitted.

That's an interesting point I hadn't considered before. Git, as a distributed vcs, takes the position that everyone can edit all files, and rare conflicts can be managed easily because changes are relatively small and diffable. But these assumptions break down with binary assets. Changes are not small, they typically change entire files at once. They're also not diffable. As a result, conflicts are not rare, they're…

Note that GitHub itself does create useful diffs of images: https://github.com/cameronmcefee/Image-Diff-View-Modes/commi... More: https://github.com/blog/817-behold-image-view-modes

Useful for seeing what changed, not so useful for merging unless the images are extremely simple.

Re: Announcing Git Large File Storage

#133

Post by neandrake »

jordigh wrote: post #20
vvanders wrote: post #14

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeable binary assets. Otherwise you get two people touching the same file and someone has to destroy their changes. That never makes anyone happy. It's also unseen how good the disk performance is. These two areas are the reason why Perforce is still my go-to solut…

Mercurial has largefiles and locking too: http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LockExtension Like other people have noticed, you can have the good parts of a DVCS and the good parts of a CVCS. It doesn't have to be either-or: https://blogs.janestreet.com/centralizing-distributed-versio... http://bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-...

We turned on largefiles extension almost a year ago and have come to regret that decision. The major pain point is with integrations, from Eclipse plugin to most any repository management/hosting solution tend to either have bugs or full-on don't support those repos. With plain mercurial workflow it works in most situations, but still rears its ugly head. The maintainers of mercurial classify it as a "feature of last resort" (Read http://mercurial.selenic.com/wiki/FeaturesOfLastResort).

Switching off largefiles requires rebuilding the repository which rebuilds the entire repository. Orchestrating the migration to a new repository for engineering department is also painful which is why we're stuck for the near future (for example ongoing support for a version that's built from largefiles repo with ongoing feature work in non-largefiles repo).

The tooling for mercurial tends to lack behind git's, likely due to git's enormous popularity - so I personally would recommend avoiding largefiles extension.

Re: Announcing Git Large File Storage

#134

Post by luckydude »

BitKeeper has had a better version of this since around 2007. Better in that we support a cloud of servers so there is no "close to the server" thing, everyone is close to the server.

What we don't have is the locking. I agree with the people commenting here that locking is a requirement because you can't merge. We need to do that.

Re: Announcing Git Large File Storage

#135

Post by sophacles »

IgorPartola wrote: post #100
gjtorikian wrote: post #83

Earlier quoted context omitted.

Yes, you can. That's one other advantage over git-annex (albeit slight). The documentation lays out the workflow: https://help.github.com/articles/configuring-large-file-stor... As does the website: https://git-lfs.github.com/ (see: "Getting Started")

Righto. So what is the equivalent thing with git-annex? Would I have to essentially do two commits, one for source code and one for large objects?

I'm still ramping up on git-annex, but basically (once set up):

    git annex add large_file
    git commit
And with mixed (haven't experimented with that yet), I'm pretty sure you could: git annex add large_file git add small_file git commit

Re: Announcing Git Large File Storage

#137

Post by durin42 »

neandrake wrote: post #133
jordigh wrote: post #20

Earlier quoted context omitted.

Mercurial has largefiles and locking too: http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LockExtension Like other people have noticed, you can have the good parts of a DVCS and the good parts of a CVCS. It doesn't have to be either-or: https://blogs.janestreet.com/centralizing-distributed-versio... http://bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-...

We turned on largefiles extension almost a year ago and have come to regret that decision. The major pain point is with integrations, from Eclipse plugin to most any repository management/hosting solution tend to either have bugs or full-on don't support those repos. With plain mercurial workflow it works in most situations, but still rears its ugly head. The maintainers of mercurial classify it as a "feature of last…

OOC, what kinds of problems did you see with editor integrations? I work on hg, and I'm wondering if there's anything we can do to make the situation less painful.

Re: Announcing Git Large File Storage

#138

Post by johnsea »

sytse wrote: post #105
zipperhead wrote: post #61

Earlier quoted context omitted.

As a counterpoint, I just set up git-annex and sync'd a couple of local servers plus a remote server, with no issues, by following along in the walkthrough. Granted, it's not exactly an out-of-the-box setup, like say, syncthing, but it wasn't anything overly difficult.

If you want to use git-annex without any setup consider using GitLab.com, it is enabled by default and free to use.

Make a Kickstarter to put it in Gitlab CE, I'd propose.

Re: Announcing Git Large File Storage

#139

Post by leni536 »

derefr wrote: post #93
WallWextra wrote: post #34

Earlier quoted context omitted.

A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.

I think the problem here is that people are jumping from "git should be able to store binary assets" to "git should be able to store binary compilation objects ." You would never check a .o file into your SCM; an SCM, as the name implies, is for managing source files, not object files. A PSD or a DOCX is also a source file, despite being binary: they're representations of the work-in-progress itself, containing enoug…

Someone could write an image editor where the project format is a directory with the source jpeg or png assets and a makefile which generates the resulting image with imagemagick (or other) calls. The makefile could be easily diffable and mergeable and people wouldn't touch the binary assets (or at least wouldn't expect to diff or merge them).

Brushes wouldn't be easy though. This could be handled by trivial svg or ps paths which would get rendered in the make process. This way even brush strokes could be diffable. I wounder if it would be possible to convert existing PSD files or Gimp project files into such a format.

Re: Announcing Git Large File Storage

#140

Post by icebraining »

sophacles wrote: post #135
IgorPartola wrote: post #100

Earlier quoted context omitted.

Righto. So what is the equivalent thing with git-annex? Would I have to essentially do two commits, one for source code and one for large objects?

I'm still ramping up on git-annex, but basically (once set up): git annex add large_file git commit And with mixed (haven't experimented with that yet), I'm pretty sure you could: git annex add large_file git add small_file git commit

Yes, you can mix them just fine.
Post reply on HN