I'm still unconvinced. Will this provide a user friendly approach to managing design assets.
Git partial clone lets you fetch only the large file you need
31–40 of 88 posts
Re: Git partial clone lets you fetch only the large file you need
#32This could actually be a really good solution to the maximum supported size of a Go module. If you place a go.mod in the root of your repo, then every file in the repo becomes part of the module. There's also a hardcoded maximum size for a module: 500M. Problem is, I've got 1G+ of vendored assets in one of my repos. I had to trick Go into thinking that the vendored assets were a different Go module[0]. Go would have to add support for this, but it would be a pretty elegant solution to the problem.
Re: Git partial clone lets you fetch only the large file you need
#33Earlier quoted context omitted.
The problem with git submodules is they can't be used like a hyperlink to another repository. Updating the submodule requires updating the superproject as well. The new commits are invisible to the superproject until that is done. It'd be great if they worked like Python's editable package installations.
Then the state of the superproject would depend on when the checkout occurred. That would be disastrous for consistency, you’d be unable to replicate a checkout later or elsewhere. The state of a repo after a checkout should only depend on the commit that was checked out.
It’s interesting that we’ve never developed the equivalent for Git of what every programming-language ecosystem has: keeping two parallel listings of dependencies, one in terms of version constraints to satisfy, and the other in terms of exact refs.
I could totally see a .gitmodules.reqs file specified in terms of semver specs against tags, or just listing a branch to check out the HEAD of; resolving to the same .gitmodules file we already have. Not even a breaking change!
Re: Git partial clone lets you fetch only the large file you need
#34I'm still unconvinced. Will this provide a user friendly approach to managing design assets.
My impression is that it will use the normal git experience managing design assets. Ie. with this there should be no need for additional tooling. If it works, that would be so great!
Re: Git partial clone lets you fetch only the large file you need
#35This could actually be a really good solution to the maximum supported size of a Go module. If you place a go.mod in the root of your repo, then every file in the repo becomes part of the module. There's also a hardcoded maximum size for a module: 500M. Problem is, I've got 1G+ of vendored assets in one of my repos. I had to trick Go into thinking that the vendored assets were a different Go module[0]. Go would have…
That does sound like a "you're holding it wrong" issue. As one of the Go team members pointed out, defining a separate module is not a hack, but the intended way of doing it.
How would a partial checkout help?
Re: Git partial clone lets you fetch only the large file you need
#36Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.
I’ve done that. Especially if you want specific versions of data to build ML models, this makes a nice audit log for reproducibility
Re: Git partial clone lets you fetch only the large file you need
#37Earlier quoted context omitted.
Then the state of the superproject would depend on when the checkout occurred. That would be disastrous for consistency, you’d be unable to replicate a checkout later or elsewhere. The state of a repo after a checkout should only depend on the commit that was checked out.
This is great for vendoring external dependencies that aren't under the developer's control. When the same developer is working on several related but separate projects at the same time, it's too cumbersome. Would be nice if git submodules could also point to a branch instead of specific commits. That way, the superproject's state would not be modified every time the branch is updated.
If your software is small enough that a small handful for engineers can keep the whole thing in their head, you probably don't need submodules. If you have different teams working on different parts of the project, submodules start making more sense.
Re: Git partial clone lets you fetch only the large file you need
#38This could actually be a really good solution to the maximum supported size of a Go module. If you place a go.mod in the root of your repo, then every file in the repo becomes part of the module. There's also a hardcoded maximum size for a module: 500M. Problem is, I've got 1G+ of vendored assets in one of my repos. I had to trick Go into thinking that the vendored assets were a different Go module[0]. Go would have…
That does sound like a "you're holding it wrong" issue. As one of the Go team members pointed out, defining a separate module is not a hack, but the intended way of doing it. How would a partial checkout help?
Go modules are built around git, unlike many other languages package systems. That means you don't get to pick and choose what goes into them. Imagine if you had to put an empty package.json in every (non-node) directory of your git repo to exclude it from an NPM package, or an install.py in every (non-python) directory to exclude it from a PyPI package. Multi-language repos would get ridiculous pretty quickly.
Re: Git partial clone lets you fetch only the large file you need
#39Also --reference (or --shared) is a good parameter to speed-up cloning (for build, for example), if you have your repository cached in some other place. I was using it a long time ago when I was working on system that required to clone 20-40 repos to build. This approach decreased clone timings by an order of magnitude.
Do you actually need clones in that scenario? I worked on a build system that grabbed source from several hundred repos at the starting point, and it turned out to be way faster to just grab it all as tarballs with aria2c.
Grapping the tarbell from where? To my best knowledge, tarbell export is not a part of git, but something git hosts provide.
Git is a distributed VCS, and we should support keeping it that way.
Re: Git partial clone lets you fetch only the large file you need
#40Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.
This kind of comment isn't helpful. Of course, there have been ways to copy large files around since there were networks. What's new in this protocol enhancement is that this works within the context of a Merkle tree-based technology (upon which all DVCS's are based). To use your analogy, yes this is a wheel but it's built with rubber instead of wood and iron.
I guess I should have expanded more.
DVCS is in direct opposition of workflows that include binary files(yes I'm aware that git lfs has locking, it's also centrally orchestrated) because you can't merge almost every binary format.
We were using P4 ~15 years ago for these workflows and rather than understanding what made them work people are just rediscovering the same problems that have already been solved.
My guess is that we'll next see a solution that dynamically caches most downloaded files in a geographic friendly way, heck we may even call it "P4Proxy".
I've seen so much FUD around how git is the "one true workflow" because other solutions "don't scale" when they don't understand the constraints that certain workflows impose. Git/DVCS is great for a lot of things but sometimes you should use the right tool for the job rather than hack something together.
[Edit] These reasons are exactly why you see Unreal supporting P4/SVN out of the box[1] and no mention of git.
[1] https://docs.unrealengine.com/en-US/Engine/UI/SourceControl/...