Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git partial clone lets you fetch only the large file you need

about.gitlab.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Git partial clone lets you fetch only the large file you need

#51

Post by beagle3 »

There is one note piece to the puzzle to make git perfect for every use case I can think of: store large files as a list of blobs broken down by some rolling hash a-la rsync/borg/bup.

That would e.g. make it reasonable to check in virtual machine images or iso images into a repository. Extra storage (and by extension, network bandwidth) would be proportional to change size.

git has delta compression for text as an optimization but it’s not used on big binary files and is not even online (only on making a pack). This would provide it online for large files.

Junio posted a patch that did that ages ago, but it was pushed back until after the sha1->sha256 extension.

Re: Git partial clone lets you fetch only the large file you need

#52

Post by lima »

jniedrauer wrote: post #38
lima wrote: post #35

Earlier quoted context omitted.

That does sound like a "you're holding it wrong" issue. As one of the Go team members pointed out, defining a separate module is not a hack, but the intended way of doing it. How would a partial checkout help?

Go modules are built around git, unlike many other languages package systems. That means you don't get to pick and choose what goes into them. Imagine if you had to put an empty package.json in every (non-node) directory of your git repo to exclude it from an NPM package, or an install.py in every (non-python) directory to exclude it from a PyPI package. Multi-language repos would get ridiculous pretty quickly.

This is only an issue if you put a go.mod file at the top level of your repo. We have monorepos with hundreds of modules.

Re: Git partial clone lets you fetch only the large file you need

#53

Post by dagmx »

maccard wrote: post #50
dagmx wrote: post #44

Earlier quoted context omitted.

I think this is a very p4 centric view of the world. Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there. The issue is neither is a silver bullet for the others workflow and needs, and they both su…

> but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Because there's 300+ people working on a project, and it's not feasible to know what every other person is working on or planning on working on. The file lock (code can be merged too, just like git, so this is only really for binary assets) is a crude communication tool saying "hey I'm using this fil…

Meanwhile film studios go on with 300+ people without hitting the issue of people hitting the same asset files without needing locking.

I think locking is a fine utility to have but I think a lot of workflows use it to workaround poor communication.

And I think you're constraining your views of git to just your workflow.

I've worked in a lot of scenarios where you need multiple remotes such as having an internal repo and an external one.

And similarly there are lots of scenarios where having a decentralized copy of the report is very useful for being able to work in offline scenarios and compare multiple branches. Things like when commuting on a plane or being in low connection areas.

I don't see how my view is git centric. I'm saying each VCS has very useful areas and equally big rough spots. The problem is that each VCS group believes there's is the only right system.

Re: Git partial clone lets you fetch only the large file you need

#54

Post by dagmx »

vvanders wrote: post #46
dagmx wrote: post #44

Earlier quoted context omitted.

I think this is a very p4 centric view of the world. Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there. The issue is neither is a silver bullet for the others workflow and needs, and they both su…

If by P4 centric you also mean SVN as well then sure. I will say however that if you think locking is optional then you already don't understand these workflows and why they're so critical. Art/design/animation doesn't care that "they should not have been touching the file" they just care that they have to throw away two days of work because someone made multiple edits to the same package file. I've literally seen mu…

I've worked as a pipeline supervisor at one of the biggest VFX studios. I very much understand these workflows. I've worked as an artist and a tech artist in perforce and SVN workflows too. I've had to support the workflows of a 1000+ workforce across multiple locations.

I don't think attributing my disagreement with you to not understanding workflows is a fair characterization.

I still think locking , while useful, is only mandatory for work cultures with poor communication. Otherwise, many companies get by with very large workforces who don't hit these issues without having locking.

Separation of code and art assets also don't need to be painful. It's very doable but does require some amount of architectural consideration.

And I very much acknowledge there are projects where code isn't the majority makeup, which is why I say that none of the VCS systems cover mixed projects well or cover all the needs of the others well.

Re: Git partial clone lets you fetch only the large file you need

#55

Post by sjburt »

derefr wrote: post #33
sjburt wrote: post #20

Earlier quoted context omitted.

Then the state of the superproject would depend on when the checkout occurred. That would be disastrous for consistency, you’d be unable to replicate a checkout later or elsewhere. The state of a repo after a checkout should only depend on the commit that was checked out.

It’s interesting that we’ve never developed the equivalent for Git of what every programming-language ecosystem has: keeping two parallel listings of dependencies, one in terms of version constraints to satisfy, and the other in terms of exact refs. I could totally see a .gitmodules.reqs file specified in terms of semver specs against tags, or just listing a branch to check out the HEAD of; resolving to the same .git…

It would mean attaching a semantic meaning to tags, but git doesn't do that, ever, for any reference. You don't even have to have a master branch, much less tags that follow semver. Linux doesn't even use semver!

Re: Git partial clone lets you fetch only the large file you need

#56

Post by vvanders »

dagmx wrote: post #54
vvanders wrote: post #46

Earlier quoted context omitted.

If by P4 centric you also mean SVN as well then sure. I will say however that if you think locking is optional then you already don't understand these workflows and why they're so critical. Art/design/animation doesn't care that "they should not have been touching the file" they just care that they have to throw away two days of work because someone made multiple edits to the same package file. I've literally seen mu…

I've worked as a pipeline supervisor at one of the biggest VFX studios. I very much understand these workflows. I've worked as an artist and a tech artist in perforce and SVN workflows too. I've had to support the workflows of a 1000+ workforce across multiple locations. I don't think attributing my disagreement with you to not understanding workflows is a fair characterization. I still think locking , while useful,…

I think we'll just have to agree to disagree.

It sounds like we just come from different development cultures. Your solution to lack of locking sounds like a top-down hierarchy that wouldn't be flexible enough to support the teams I've worked with.

Having seen both approaches(and how they break down) I'll take a centralized locking solution over communication mistakes that lead to days of work being lost.

Re: Git partial clone lets you fetch only the large file you need

#57

Post by phaemon »

sewer_bird wrote: post #3
piliberto wrote: post #2

> One reason projects with large binary files don't use Git is because, when a Git repository is cloned, Git will download every version of every file in the repository. Wrong? There's a --depth option for the git fetch command which allows the user to specify how many commits they want to fetch from the repository

Yes, but 95% of devs, even fairly talented ones, don't really know how to use Git.

Git is fundamentally very simple. Any dev who doesn't understand exactly how it works is not even remotely talented.

Re: Git partial clone lets you fetch only the large file you need

#58

Post by pas »

beagle3 wrote: post #51

There is one note piece to the puzzle to make git perfect for every use case I can think of: store large files as a list of blobs broken down by some rolling hash a-la rsync/borg/bup. That would e.g. make it reasonable to check in virtual machine images or iso images into a repository. Extra storage (and by extension, network bandwidth) would be proportional to change size. git has delta compression for text as an op…

Do ISOs and other large blob types support only partial (block) modification? Wouldn't all subsequent blocks change too?

Re: Git partial clone lets you fetch only the large file you need

#59

Post by hinkley »

jniedrauer wrote: post #30
derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

Using submodules require that everyone on your team has at least a vague idea of what's going on and how to not foot-gun themselves. That's hard enough with git itself. I don't think I've ever seen submodules used without become a major pain point.

That is a straight up nonstarter.

Someone was trying to talk me into git subtrees though...

Re: Git partial clone lets you fetch only the large file you need

#60

Post by dijit »

vvanders wrote: post #19

Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.

> Only downside of course is that it isn't free.

Another downside is that it consumes insane resources (our servers are in the dozens of TiB of ram, with huge NVMe based storage arrays directly attached)

Another downside is that you have to maintain connection to p4 to do any VCS operations (stashing included).

Another downside is that branches are very "expensive" (often taking days) and are impossible to reconcile. We never re-merge to MAIN.

Post reply on HN