Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git partial clone lets you fetch only the large file you need

about.gitlab.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Git partial clone lets you fetch only the large file you need

#71

Post by dagmx »

adrianmonk wrote: post #69
dagmx wrote: post #44

Earlier quoted context omitted.

I think this is a very p4 centric view of the world. Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there. The issue is neither is a silver bullet for the others workflow and needs, and they both su…

> Why are people even touching files they shouldn't be touching? I don't think it's accurate to say they shouldn't be touching the files. It's possible to make two unrelated changes in the same binary file just as it is possible to make two unrelated changes in a source file (or other merge-friendly text file). Just as there may be nothing wrong if one person changes the function foo() in a file and someone else chan…

Sure, but I would argue it's a poor system that is setup to require serial editing to a non diffable file.

Again, I'm not saying locking is an invalid solution. It is. But to me, it's often (but not always), a crutch for a deeper issue.

That binary file should be set up to be modular if it is intended to have areas that multiple users can touch without directly affecting each other.

Re: Git partial clone lets you fetch only the large file you need

#72

Post by vvanders »

dagmx wrote: post #70
vvanders wrote: post #67

Earlier quoted context omitted.

What happens if you're not around to drive the process? What about if you don't have the organizational backing to drive the process? What if a team goes AWOL or isn't bought in to your process? I've seen variants of all those happen in production in one form or another. At the end of the day humans make mistakes, especially when involving communication. I'd rather have a physical system that prevents breaks instead…

It's very rare that someone needs to be around to oversee the process. Tooling guides the vast majority of users in to a workflow that works while still being flexible should they need it. Lack of organizational backup, do you mean cultural from the studio or infrastructure? Both are a problem no matter what solution you pick. If a team goes AWOL, that's on them. The tooling usually allows for some amount of arbitrar…

Oh I've been working with git for ~9 years now, it's not a lack of familiarity.

Take AOSP, even Google had to overlay the repo[1] tool to scale past git. It's a hot pile of garbage that won't let you sync all repos to a specific point in time. Not to mention the nature of cross-repo commits are not atomic. Good luck bisecting a breaking change across millions of lines of code and build files.

I've spent over a week chasing down how some homespun tool for storing binary assets side-by-side with git works so I could get a single file into a build.

Last company I was at which was a leader in the Android space just put the whole thing in P4, branch per device and it worked without many major issues. Pulling source took 1/50th the time a repo sync took. Literally an A/B comparison of one tech vs the other. That's before you even start to consider prebuilts.

Like I said, I think we're just going to have to agree to disagree and leave it at that.

[1] https://gerrit.googlesource.com/git-repo/

Re: Git partial clone lets you fetch only the large file you need

#73

Post by bspammer »

nikivi wrote: post #16

Is it possible given a git repo (hosted on say GitHub) to only 'clone' (download) certain files from it? Without `.git`

Short answer is, not easily: https://stackoverflow.com/a/14610427

You can get the most recent tree for a repository (no history, just the current state of the repo) with `git clone --depth=1`. That's often good enough for slow connections.

Re: Git partial clone lets you fetch only the large file you need

#74

Post by yencabulator »

jniedrauer wrote: post #38
lima wrote: post #35

Earlier quoted context omitted.

That does sound like a "you're holding it wrong" issue. As one of the Go team members pointed out, defining a separate module is not a hack, but the intended way of doing it. How would a partial checkout help?

Go modules are built around git, unlike many other languages package systems. That means you don't get to pick and choose what goes into them. Imagine if you had to put an empty package.json in every (non-node) directory of your git repo to exclude it from an NPM package, or an install.py in every (non-python) directory to exclude it from a PyPI package. Multi-language repos would get ridiculous pretty quickly.

> Go modules are built around git

Not really. Modules are specced based on zip files and metadata in text files. There's just support for extracting that data from git repos transparently.

Here's a slightly out of date write-up: https://research.swtch.com/vgo-module

Re: Git partial clone lets you fetch only the large file you need

#75

Post by hinkley »

microtherion wrote: post #64

That seems quite useful, though Git LFS mostly does the job. One of my biggest remaining pain points is resumable clone/fetch. I find it near impossible to clone large repos (or fetch if there were lots of new commits) over a slow, unstable link, so almost always I end up cloning a copy to a machine closer to the repo, and rsyncing it over to my machine.

What’s your take on this line?

> Partial Clone is a new feature of Git that replaces Git LFS and makes working with very large repositories better by teaching Git how to work without downloading every file.

Re: Git partial clone lets you fetch only the large file you need

#76

Post by damnyou »

sjburt wrote: post #55
derefr wrote: post #33

Earlier quoted context omitted.

It’s interesting that we’ve never developed the equivalent for Git of what every programming-language ecosystem has: keeping two parallel listings of dependencies, one in terms of version constraints to satisfy, and the other in terms of exact refs. I could totally see a .gitmodules.reqs file specified in terms of semver specs against tags, or just listing a branch to check out the HEAD of; resolving to the same .git…

It would mean attaching a semantic meaning to tags, but git doesn't do that, ever, for any reference. You don't even have to have a master branch, much less tags that follow semver. Linux doesn't even use semver!

Correct, this feature should be built on top of the source control system, not as part of it.

Re: Git partial clone lets you fetch only the large file you need

#77

Post by saagarjha »

mikepurvis wrote: post #45
madsbuch wrote: post #39

Earlier quoted context omitted.

Grapping the tarbell from where? To my best knowledge, tarbell export is not a part of git, but something git hosts provide. Git is a distributed VCS, and we should support keeping it that way.

Almost any project you work on will have an authoritative copy of the repo in some kind of web-accessible tool, most of which provide a tarball-download function. And GitHub's scheme is pretty much a de-facto standard at this point—GitLab's implementation is an exact copy of it, for example: https:// / /archive/ .tar.gz Edit to add: Also, git-archive --remote is actually most of the way there, but it's not an HTTP do…

GitHub doing something one way and GitLab copying it doesn't make a standard.

Re: Git partial clone lets you fetch only the large file you need

#78

Post by jasondclinton »

vvanders wrote: post #72
dagmx wrote: post #70

Earlier quoted context omitted.

It's very rare that someone needs to be around to oversee the process. Tooling guides the vast majority of users in to a workflow that works while still being flexible should they need it. Lack of organizational backup, do you mean cultural from the studio or infrastructure? Both are a problem no matter what solution you pick. If a team goes AWOL, that's on them. The tooling usually allows for some amount of arbitrar…

Oh I've been working with git for ~9 years now, it's not a lack of familiarity. Take AOSP, even Google had to overlay the repo[1] tool to scale past git. It's a hot pile of garbage that won't let you sync all repos to a specific point in time. Not to mention the nature of cross-repo commits are not atomic. Good luck bisecting a breaking change across millions of lines of code and build files. I've spent over a week c…

Hi, I work at Google and replied to you way up-thread. I built a CI system used by ChromeOS based on Repo and even contributed some changes to it. While I don't like it much, it is useful. You misunderstand or are misinformed about many aspects of it.

> Google had to overlay the repo[1] tool to scale past git

It was created to allow for a forest of git repos to all coexist in a world in which git submodules wasn't suitable yet OR the repos spanned security domains. However, almost all of the shortcomings of submodules have been addressed and so—at least—the team that I lead now is considering migration to it from Repo.

> cross-repo commits are not atomic

Yes, that is a feature. But I think you meant that there's no cross-repo coordinate in the timeline to sync to. However, there is. That's exactly what a Repo tool manifest snapshot is. Our CI system ensured that change that had deps across repos were committed and a Repo manifest snapshot only was taken with all inter-commit deps satisfied.

> Good luck bisecting a breaking change across millions of lines of code and build files

The team that I led implemented this. We simply snapshotted the forest at every T time intervals. For bisection, we walked the snapshots. Once a specific manifest snapshot was identified as the culprit, we further bisected within repos for a specific change.

> Pulling source took 1/50th the time a repo sync took.

Yes, that's what the partial clones (the article you replied to) and sparse checkouts solves. Once these two things are widely available, I don't see any benefits to P4 remaining.

Re: Git partial clone lets you fetch only the large file you need

#79

Post by pas »

beagle3 wrote: post #61
pas wrote: post #58

Earlier quoted context omitted.

Do ISOs and other large blob types support only partial (block) modification? Wouldn't all subsequent blocks change too?

Sometimes they do - e.g. if you replace a file in the ISO that is the same size up to block alignment, which is common when e.g. editing a text file or recompiling an executable with a minor change. They almost always do when it's a VM image representing a disk - only some blocks change every write. However, with self synchronizing hashes of the kind used by rsync bup and borg, it doesn't matter - you could have a 1T…

Oh, I used rsync many times but I thought it simply retransmits changed files. (Oh, it needs the --checksum argument to do this, okay.)

So how do these self-synchronizing hashes work? Like a Merkle Tree? (Ah, okay https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_... )

So rsync uses 8KB for chunk size, so for a file 1GB it has 125 000 chunks. (And if every chunk needs 16 bytes of hash data to send, that's about 2MB, pretty darn efficient, especially if it can spot reorders.) Though according to Wikipedia it only does this if the target file has the same size, so adding new files to ISOs might not work in case of rsync, but still, the possibility is there for diff algos and version control systems.

Re: Git partial clone lets you fetch only the large file you need

#80

Post by pas »

adrianmonk wrote: post #66
pas wrote: post #58

Earlier quoted context omitted.

Do ISOs and other large blob types support only partial (block) modification? Wouldn't all subsequent blocks change too?

It really depends on the type of file. ("Other large blob types" is a rather broad category.) One obvious example where you could have a lot of common blocks (even following the offset where a change was made) is zip files. The zip format basically compresses each file individually and then concatenates all that together. Let's say you have a build and it packages the results up as a big zip file. (Java builds often…

So ZIPs don't have any "global" directory thing? :o
Post reply on HN