Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Committing changes to a 130GB Git repository without full checkouts [video]

youtube.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#11

Post by korijn »

eliomattia wrote: post #10
korijn wrote: post #8

Earlier quoted context omitted.

We wrote our own LFS API server (which is actually not very hard, about 100 lines of python was enough and it performs at scale) so we can directly leverage azure blob storage. If you don't walk this path and enable LFS in github or something like that the costs are obscene, yes. For us it's dirt cheap. If I check out head of my repo and don't filter anything, it's a couple GBs. Inside the azure blob storage containe…

That is really interesting and begs the question of how frequently you have changes in your data that lead to new commits. I am assuming here that you don't dedupe anything, that is, you throw the entire files into Azure with each version, since it's cheap enough for your purposes. Also, how frequently do you move head, even without committing anything new, perhaps to use another branch?

LFS stores files by content hash, so deduplication happens that way. But you're right that if you frequently make small changes to a single large file, it's wasteful.

In our case though we don't frequently change files, we just get lots and lots of new big files coming in all the time.

Moving head, as in, to check out another branch locally? Somewhat regularly I guess. I suppose you're wondering about performance in that scenario? It's usually quite good since git-lfs does some local caching as well. I've never needed to wait longer than a couple of seconds. I'm usually on a wired 1000/1000 Mbit optic fibre connection, and transfers are directly to and from an azure blob storage container (the LFS API server only generates download and upload URLs, it intentionally doesn't transfer any data), with parallel connections and chunking etc, so it doesn't really get any better than that. And all of that is out of the box functionality too. :)

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#12

Post by eliomattia »

korijn wrote: post #11
eliomattia wrote: post #10

Earlier quoted context omitted.

That is really interesting and begs the question of how frequently you have changes in your data that lead to new commits. I am assuming here that you don't dedupe anything, that is, you throw the entire files into Azure with each version, since it's cheap enough for your purposes. Also, how frequently do you move head, even without committing anything new, perhaps to use another branch?

LFS stores files by content hash, so deduplication happens that way. But you're right that if you frequently make small changes to a single large file, it's wasteful. In our case though we don't frequently change files, we just get lots and lots of new big files coming in all the time. Moving head, as in, to check out another branch locally? Somewhat regularly I guess. I suppose you're wondering about performance in…

Sorry I should have been more specific, I meant block deduplication, or any form of deduplication at a level lower than the entire file. File deduplication can only get you so far, depending on the use case. XetHub does block deduplication, whereas I am implementing data-level deduplication, which is slower in recreating dataset snapshots (can be parallelized and delegated), but allows savings on disk space with small but frequent changes and can be tied to collaborative features to show diffs, comment on them, and revert or edit changes where needed, all while pointing clearly to specific commits. And also potentially fork data or cumulative changes.

Yes I meant either checking out other branches locally, or in the general case pointing to another branch to indicate to any services to make data from that branch available to wherever it's consumed. I am assuming that each incoming new file is then added to data pipelines, possibly just a few. Sounds like you are in the sweet spot where you have the speed you want and, given unfrequent changes, you are fine with the versions taking up terabytes on Azure, since they are mostly new data.

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#13

Post by eliomattia »

korijn wrote: post #5

I use git lfs. There are filter options for all commands so you don't need to checkout any more data than you want to/need to. Works like a charm for me! I'd be curious to hear what features you are missing. We have repositories that would be as big as 100GB if you downloaded all large files for the full history, but I guess I don't see why you would want do that?

Just read the second paragraph. Currently expanding merge resolution assistance to deal with the general merge conflict case, as well as implementing revert and cherry-pick assistance. Unsure if that is what you were wondering? You probably don't want to do that with 100 GB if most of your commits are new data, rather than changes, yet I wonder whether all incoming new files are then queued into the same pipelines and the reason why they are separate files is not to have to deal with one giant cumulative file for which the older parts would not be deduped in Git LFS, which is a great reason, or whether those files are anyway different file types in terms of contents and intended use, another great reason, or something else altogether. Are you processing data by including anything in a given folder in a given commit?

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#14

Post by KingLancelot »

eliomattia wrote: post #9
KingLancelot wrote: post #3

Honestly, it’d be nice if there was like a PNGCrush for git repos. Or maybe even if Git offered zstd compression would be cool too.

Git does do compression on repos, but the fact that versioning repositories with (huge) data is still an open problem suggests that it is not the kind that fixes it. I might be mistaken, are you aware of any interesting compression methods applied to version control?

I know git uses deflate hence my first paragraph.

Doesn’t mean those deflate trees are optimal, as you can see with tools like OxiPNG optimizing the deflate compression to reduce png file sizes by about half.

The same optimization could be applied to git blobs in theory, it would be cool if there was a tool that did that.

——

My second point was more about if git was upgraded to add a new compression algorithm like zstd instead of Deflate, like switching the hash algorithm from SHA-1 to SHA-2 (though if I was in charge, I’d go with Blake3 because it’s far faster)

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#16

Post by eliomattia »

scrubs wrote: post #15

Where do git sparse check outs stop and this begin?

The two are not mutually exclusive, in principle. Depending on workflows, sizes, and change frequencies, each has advantages. Sparse checkouts are useful with small files that specific individuals can focus upon, among other things. This is for keeping server-side repo size small in versioning larger datasets that change daily and for avoiding the requirement to have a sparse checkout of those large datasets in workflows with incoming data, in the first place. This is an interesting use case: https://news.ycombinator.com/item?id=35763004

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#17

Post by eliomattia »

eliomattia wrote: post #13
korijn wrote: post #5

I use git lfs. There are filter options for all commands so you don't need to checkout any more data than you want to/need to. Works like a charm for me! I'd be curious to hear what features you are missing. We have repositories that would be as big as 100GB if you downloaded all large files for the full history, but I guess I don't see why you would want do that?

Just read the second paragraph. Currently expanding merge resolution assistance to deal with the general merge conflict case, as well as implementing revert and cherry-pick assistance. Unsure if that is what you were wondering? You probably don't want to do that with 100 GB if most of your commits are new data, rather than changes, yet I wonder whether all incoming new files are then queued into the same pipelines an…

*I have just (only now) read the second paragraph in your message. Not sure if that came across correctly, that first sentence was too compressed.

Re: Committing changes to a 130GB Git repository without full checkouts [video]

#18

Post by eliomattia »

KingLancelot wrote: post #14
eliomattia wrote: post #9

Earlier quoted context omitted.

Git does do compression on repos, but the fact that versioning repositories with (huge) data is still an open problem suggests that it is not the kind that fixes it. I might be mistaken, are you aware of any interesting compression methods applied to version control?

I know git uses deflate hence my first paragraph. Doesn’t mean those deflate trees are optimal, as you can see with tools like OxiPNG optimizing the deflate compression to reduce png file sizes by about half. The same optimization could be applied to git blobs in theory, it would be cool if there was a tool that did that. —— My second point was more about if git was upgraded to add a new compression algorithm like zs…

Fully agree. Compression in many cases removes the ability to diff easily, however. In a large dataset where, in terms of size, 1% of the original data undergoes changes, or new data the size of 1% of the original dataset is added, I think compressing does not compare with just deduplicating the unchanged 99% in terms of storage, but when speed is the #1 factor, the discussion is more nuanced. It might be interesting to have a combination of deduplication and better compression of the changes, in some form, to get the optimal tradeoff. Repo sizes in ML these days are high, I'm curious which repository compression techniques are being evaluated and deployed.
Post reply on HN