Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git partial clone lets you fetch only the large file you need

about.gitlab.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Git partial clone lets you fetch only the large file you need

#11

Post by ComputerGuru »

matheusmoreira wrote: post #9
derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

The problem with git submodules is they can't be used like a hyperlink to another repository. Updating the submodule requires updating the superproject as well. The new commits are invisible to the superproject until that is done. It'd be great if they worked like Python's editable package installations.

They can now, with the new-ish submodule update/init --remote. But the problem with sub modules is that you cannot do a shallow fetch (depth 1) because most hosts won’t serve unadvertised refs.

Re: Git partial clone lets you fetch only the large file you need

#12

Post by ComputerGuru »

piliberto wrote: post #2

> One reason projects with large binary files don't use Git is because, when a Git repository is cloned, Git will download every version of every file in the repository. Wrong? There's a --depth option for the git fetch command which allows the user to specify how many commits they want to fetch from the repository

depth is broken; it cannot be used for submodules/recursive submodules dependably because most hosts will refuse to serve unadvertised refs. We learned this the hard way. Or maybe it is submodules that are broken. Or git itself.

Re: Git partial clone lets you fetch only the large file you need

#13

Post by elephantum »

shaklee3 wrote: post #4

This is great. We use get lfs extensively, and one of the biggest complaints we have is users have to clone 7GB of data just to get the source files. There's a work around in that you don't have to enter your username and password from the lfs repo, and let it timeout, but that's a kluge.

There’s an option for that: GIT_LFS_SKIP_SMUDGE=1 git clone SERVER-REPOSITORY

Re: Git partial clone lets you fetch only the large file you need

#14

Post by mikepurvis »

itroot wrote: post #8

Also --reference (or --shared) is a good parameter to speed-up cloning (for build, for example), if you have your repository cached in some other place. I was using it a long time ago when I was working on system that required to clone 20-40 repos to build. This approach decreased clone timings by an order of magnitude.

Do you actually need clones in that scenario? I worked on a build system that grabbed source from several hundred repos at the starting point, and it turned out to be way faster to just grab it all as tarballs with aria2c.

Re: Git partial clone lets you fetch only the large file you need

#15

Post by [deleted] »

derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

[deleted]

Re: Git partial clone lets you fetch only the large file you need

#17

Post by fizixer »

nikivi wrote: post #16

Is it possible given a git repo (hosted on say GitHub) to only 'clone' (download) certain files from it? Without `.git`

I believe you're looking for the 'working tree' only. You could do the following:

git archive --remote= | tar -t

source: https://stackoverflow.com/questions/3946538

Re: Git partial clone lets you fetch only the large file you need

#18

Post by scarecrow112 »

This is interesting and could be a savior for Machine Learning(ML) engineering teams. In a typical ML workflow, there are three main entities to be managed: 1. Code 2. Data 3. Models Systems like Data Version Control(DVC) [1], are useful for versioning 2 & 3. DVC improves on usability by residing inside the project's main git repo while maintaining versions of the data/models that reside on a remote. With Git partial clone, it seems like the gap could still be reduced between 1 & 2/3.

[1] - https://dvc.org/

Re: Git partial clone lets you fetch only the large file you need

#19

Post by vvanders »

Also known as workspace views in P4.

It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned.

Only downside of course is that it isn't free.

Re: Git partial clone lets you fetch only the large file you need

#20

Post by sjburt »

matheusmoreira wrote: post #9
derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

The problem with git submodules is they can't be used like a hyperlink to another repository. Updating the submodule requires updating the superproject as well. The new commits are invisible to the superproject until that is done. It'd be great if they worked like Python's editable package installations.

Then the state of the superproject would depend on when the checkout occurred. That would be disastrous for consistency, you’d be unable to replicate a checkout later or elsewhere. The state of a repo after a checkout should only depend on the commit that was checked out.
Post reply on HN