Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git partial clone lets you fetch only the large file you need

about.gitlab.com

Post reply on HN
Print view View original

21–30 of 88 posts

Re: Git partial clone lets you fetch only the large file you need

#21

Post by jasondclinton »

vvanders wrote: post #19

Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.

This kind of comment isn't helpful. Of course, there have been ways to copy large files around since there were networks. What's new in this protocol enhancement is that this works within the context of a Merkle tree-based technology (upon which all DVCS's are based). To use your analogy, yes this is a wheel but it's built with rubber instead of wood and iron.

Re: Git partial clone lets you fetch only the large file you need

#22

Post by calvinlh »

nikivi wrote: post #16

Is it possible given a git repo (hosted on say GitHub) to only 'clone' (download) certain files from it? Without `.git`

If you only want a subset of the repo's files, you can use Github's Subversion interface: https://stackoverflow.com/a/18194523

Re: Git partial clone lets you fetch only the large file you need

#23

Post by chrismorgan »

ComputerGuru wrote: post #12
piliberto wrote: post #2

> One reason projects with large binary files don't use Git is because, when a Git repository is cloned, Git will download every version of every file in the repository. Wrong? There's a --depth option for the git fetch command which allows the user to specify how many commits they want to fetch from the repository

depth is broken; it cannot be used for submodules/recursive submodules dependably because most hosts will refuse to serve unadvertised refs. We learned this the hard way. Or maybe it is submodules that are broken. Or git itself.

Depth, submodules and multiple work trees are all half-baked features that work fine up to a point, then start falling over frantically—most notably if you try to use them together.

Re: Git partial clone lets you fetch only the large file you need

#24

Post by danbolt »

In the AAA games industry git has been a bit slower on the uptake (although that’s changing quickly) as large warehouses of data are often required (eg: version history of video files, 3D audio, music, etc.). It’s nice to see git have more options for this sort of thing.

Re: Git partial clone lets you fetch only the large file you need

#25

Post by jfkebwjsbx »

danbolt wrote: post #24

In the AAA games industry git has been a bit slower on the uptake (although that’s changing quickly) as large warehouses of data are often required (eg: version history of video files, 3D audio, music, etc.). It’s nice to see git have more options for this sort of thing.

Git LFS has been a thing for years, though.

Re: Git partial clone lets you fetch only the large file you need

#26

Post by jfkebwjsbx »

derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

Submodules are almost always the wrong answer. If you need to version huge files, use Git LFS.

Re: Git partial clone lets you fetch only the large file you need

#27

Post by jfkebwjsbx »

vvanders wrote: post #19

Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.

P4 was great for its time if you could pay for it, but it is definitely not a competitor anymore.

Git LFS has been just fine for multiterabyte repositories for years.

Re: Git partial clone lets you fetch only the large file you need

#28

Post by krupan »

I started a project recently and for the first time ever I've wanted to keep large files in my repo. I looked into git LFS and was disappointed to learn that it requires either third party hosting or setting up a git LFS server myself. I looked into git annex and it seems decent. This, once it is ready for prime time, will hopefully be even better

Re: Git partial clone lets you fetch only the large file you need

#29

Post by matheusmoreira »

sjburt wrote: post #20
matheusmoreira wrote: post #9

Earlier quoted context omitted.

The problem with git submodules is they can't be used like a hyperlink to another repository. Updating the submodule requires updating the superproject as well. The new commits are invisible to the superproject until that is done. It'd be great if they worked like Python's editable package installations.

Then the state of the superproject would depend on when the checkout occurred. That would be disastrous for consistency, you’d be unable to replicate a checkout later or elsewhere. The state of a repo after a checkout should only depend on the commit that was checked out.

This is great for vendoring external dependencies that aren't under the developer's control. When the same developer is working on several related but separate projects at the same time, it's too cumbersome.

Would be nice if git submodules could also point to a branch instead of specific commits. That way, the superproject's state would not be modified every time the branch is updated.

Re: Git partial clone lets you fetch only the large file you need

#30

Post by jniedrauer »

derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

Using submodules require that everyone on your team has at least a vague idea of what's going on and how to not foot-gun themselves. That's hard enough with git itself. I don't think I've ever seen submodules used without become a major pain point.
Post reply on HN