Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Git Large File Storage

github.com

Post reply on HN
Print view View original

141–150 of 167 posts

Re: Announcing Git Large File Storage

#144

Post by davvid »

rakoo wrote: post #50
infogulch wrote: post #38

Earlier quoted context omitted.

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations i…

I don't think those are your scm's business. git is the stupid content tracker, it tracks whatever you push into it. If I were to make a contrived analogy, how do you know how to diff random arrays of bytes ? Where do you start, where do you stop ? How do you know that "\n" or "\r\n" is some kind of delimiter ? You put that knowledge in "diff" and in your editor, and git stores the raw array of bytes. It's the same w…

An image diff is pretty simple. If you represent an image as a vector of values, diffing just means subtracting two vectors abs(A-B) and writing out the result into a new image.

    template
    T diff(const T& a, const T& b)
    {
        auto sz = std::min(a.size(), b.size());
        T img(sz, 1);
        for (auto i = 0; i  a, b;
    a.emplace_back(1.0); b.emplace_back(0.5);
    a.emplace_back(1.0); b.emplace_back(0.0);
    a.emplace_back(0.5); b.emplace_back(0.5);

    auto img = diff(a, b);
    write_exr("filename.exr", img);
The resulting image ends up with 0.0 black in pixels that are identical and non-zero values in the pixels that differ. When you look at it in an image viewer only the portions that differ will be visible.

You often need to crank up the gain when the differences are small.

Re: Announcing Git Large File Storage

#146

Post by callum85 »

Can someone explain to me what problem this solves in layman's terms... How are version control systems are "impractical" for large files?

Or to put another way, what problems will I run into if I just commit large media files without using this?

Re: Announcing Git Large File Storage

#147

Post by ndepoel »

callum85 wrote: post #146

Can someone explain to me what problem this solves in layman's terms... How are version control systems are "impractical" for large files? Or to put another way, what problems will I run into if I just commit large media files without using this?

With distributed version control systems such as Git or Mercurial, when you clone a repository you get the entire history of that repository (or of a selected branch). This means that if you place large media files directly in the repository, then every clone will contain each and every revision of that file. In time, this will cause an enormous amount of bloat in your repository and slow work on the repository down to a crawl. Cloning a repository several dozens of gigabytes in size is no fun, I can tell you.

Centralized version control systems such as Subversion don't have this problem (or at least, to a lesser extent), because as a user you only download a single revision of each file when you check out the repository.

Extensions like git-media, git-fat and now git-lfs solve this issue by only storing references to large media files inside the Git repository, while storing the actual files elsewhere. With this, you will only download the revision of the large file that you actually need, when you need it. It's sort of a hybrid solution in-between centralized and decentralized version control.

Re: Announcing Git Large File Storage

#148

Post by amelius »

Wouldn't it be nicer if we had something like this on the level of the filesystem, instead of on the level of a version control system? Advantages would be that git and any other user-space application wouldn't need much extension, and files could be opened as if they were on the local file system.
Post reply on HN