Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git partial clone lets you fetch only the large file you need

about.gitlab.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Git partial clone lets you fetch only the large file you need

#81

Post by jfkebwjsbx »

danbolt wrote: post #65
jfkebwjsbx wrote: post #25

Earlier quoted context omitted.

Git LFS has been a thing for years, though.

You’re absolutely right, but larger developers and publishers have been slower to adopt. P4’s GUI/model is also intuitive for non-programming roles to learn and use historically compared to git, so a team with wide skills can ramp up quickly with a unified toolset. A less-technical manager gets a GUI that has versioning across changes from a multidisciplinary team. You can probably guess what inertia that has in a sp…

The problem is that you blamed Git, rather than your legacy workflows and corporate culture.

And again, you keep blaming Git here, now around a lack of intuitiveness and a lack of GUI. Again, Git has multiple GUIs around to choose from and multiple integrations with almost any editor and IDE you can think of, some meant for beginners and trivial usage.

And no, things are not "changing" and Git is not to be compared with a "web framework fad". Git became the version control system more than 5 years ago.

Re: Git partial clone lets you fetch only the large file you need

#82

Post by dagmx »

jfkebwjsbx wrote: post #81
danbolt wrote: post #65

Earlier quoted context omitted.

You’re absolutely right, but larger developers and publishers have been slower to adopt. P4’s GUI/model is also intuitive for non-programming roles to learn and use historically compared to git, so a team with wide skills can ramp up quickly with a unified toolset. A less-technical manager gets a GUI that has versioning across changes from a multidisciplinary team. You can probably guess what inertia that has in a sp…

The problem is that you blamed Git, rather than your legacy workflows and corporate culture. And again, you keep blaming Git here, now around a lack of intuitiveness and a lack of GUI. Again, Git has multiple GUIs around to choose from and multiple integrations with almost any editor and IDE you can think of, some meant for beginners and trivial usage. And no, things are not "changing" and Git is not to be compared w…

So I fully agree with you, but one area for more artist heavy workflows where git still struggles is ease of use.

The two biggest issues are: - git lfs doesn't automatically identify large files or binary files. So it's very easy for even experienced engineers to have set up lfs but forgotten to track a file or extension

- git exposes too much of its internals. It's really cumbersome for artists even with UI tools.

That's not to fault git as a technology but I think there's a place for an artist friendly layer on top of git, perhaps a very artist centric UI and set of tools and workflow guides

Re: Git partial clone lets you fetch only the large file you need

#83

Post by beagle3 »

pas wrote: post #79
beagle3 wrote: post #61

Earlier quoted context omitted.

Sometimes they do - e.g. if you replace a file in the ISO that is the same size up to block alignment, which is common when e.g. editing a text file or recompiling an executable with a minor change. They almost always do when it's a VM image representing a disk - only some blocks change every write. However, with self synchronizing hashes of the kind used by rsync bup and borg, it doesn't matter - you could have a 1T…

Oh, I used rsync many times but I thought it simply retransmits changed files. (Oh, it needs the --checksum argument to do this, okay.) So how do these self-synchronizing hashes work? Like a Merkle Tree? (Ah, okay https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_... ) So rsync uses 8KB for chunk size, so for a file 1GB it has 125 000 chunks. (And if every chunk needs 16 bytes of hash data to send, that's a…

No, target doesn’t have to be same size. As an optimization, if size and datetime are the same, rsync will assume no change and will not hash at all (though you can force it to).

But it will definitely use hashes when size differs (unless forced to copy whole files, or copying between local file systems)

Re: Git partial clone lets you fetch only the large file you need

#84

Post by beagle3 »

pas wrote: post #80
adrianmonk wrote: post #66

Earlier quoted context omitted.

It really depends on the type of file. ("Other large blob types" is a rather broad category.) One obvious example where you could have a lot of common blocks (even following the offset where a change was made) is zip files. The zip format basically compresses each file individually and then concatenates all that together. Let's say you have a build and it packages the results up as a big zip file. (Java builds often…

So ZIPs don't have any "global" directory thing? :o

They have a non essential copy of the directory at the end for spoed; tools exist to rebuild it if it is corrupted from the entries inside the file. But it is usually very small (the only real life exception I met is the hvsc archive where the directory size is very significant - so they zip it again)

Re: Git partial clone lets you fetch only the large file you need

#85

Post by vvanders »

jasondclinton wrote: post #78
vvanders wrote: post #72

Earlier quoted context omitted.

Oh I've been working with git for ~9 years now, it's not a lack of familiarity. Take AOSP, even Google had to overlay the repo[1] tool to scale past git. It's a hot pile of garbage that won't let you sync all repos to a specific point in time. Not to mention the nature of cross-repo commits are not atomic. Good luck bisecting a breaking change across millions of lines of code and build files. I've spent over a week c…

Hi, I work at Google and replied to you way up-thread. I built a CI system used by ChromeOS based on Repo and even contributed some changes to it. While I don't like it much, it is useful. You misunderstand or are misinformed about many aspects of it. > Google had to overlay the repo[1] tool to scale past git It was created to allow for a forest of git repos to all coexist in a world in which git submodules wasn't su…

None of the things you're talking about exist within the repo tool, it sounds like these are all things that you had to layer on top with a separate CI system.

It's been 12 years since the Dream was released and we're still not to the same level of perf/features as just stuffing the whole of AOSP in P4. I get that git has advantages, and it's an awesome tool when used appropriately but the desire to use it to solve every SCM problem under the sun is a bit misguided.

Re: Git partial clone lets you fetch only the large file you need

#86

Post by labawi »

pas wrote: post #80
adrianmonk wrote: post #66

Earlier quoted context omitted.

It really depends on the type of file. ("Other large blob types" is a rather broad category.) One obvious example where you could have a lot of common blocks (even following the offset where a change was made) is zip files. The zip format basically compresses each file individually and then concatenates all that together. Let's say you have a build and it packages the results up as a big zip file. (Java builds often…

So ZIPs don't have any "global" directory thing? :o

They don't have a global compression dictionary thing.

Similar effect can be achieved with gzip --rsyncable, which IIRC resets the dictionary based on a rolling sum.

Re: Git partial clone lets you fetch only the large file you need

#87

Post by danbolt »

jfkebwjsbx wrote: post #81
danbolt wrote: post #65

Earlier quoted context omitted.

You’re absolutely right, but larger developers and publishers have been slower to adopt. P4’s GUI/model is also intuitive for non-programming roles to learn and use historically compared to git, so a team with wide skills can ramp up quickly with a unified toolset. A less-technical manager gets a GUI that has versioning across changes from a multidisciplinary team. You can probably guess what inertia that has in a sp…

The problem is that you blamed Git, rather than your legacy workflows and corporate culture. And again, you keep blaming Git here, now around a lack of intuitiveness and a lack of GUI. Again, Git has multiple GUIs around to choose from and multiple integrations with almost any editor and IDE you can think of, some meant for beginners and trivial usage. And no, things are not "changing" and Git is not to be compared w…

Sorry if my tone came off as blaming like you said. I didn't mean to be blaming git or holding it responsible for something. I see a lot of existing inertia for Perforce in the AAA game development industry, and wanted to express that.

I think you're correct in that git is the go-to version control software. I'd reach for it as a default tool every time. I do work with older programmers where the majority of their careers have been in Visual C++ and Perforce, and I've definitely heard sentiments seeing it as wheel-reinventing. I don't agree with them, but it's what I've experienced.

Re: Git partial clone lets you fetch only the large file you need

#88

Post by microtherion »

hinkley wrote: post #75
microtherion wrote: post #64

That seems quite useful, though Git LFS mostly does the job. One of my biggest remaining pain points is resumable clone/fetch. I find it near impossible to clone large repos (or fetch if there were lots of new commits) over a slow, unstable link, so almost always I end up cloning a copy to a machine closer to the repo, and rsyncing it over to my machine.

What’s your take on this line? > Partial Clone is a new feature of Git that replaces Git LFS and makes working with very large repositories better by teaching Git how to work without downloading every file.

I believe partial clone makes the situation a little better, but it's not nearly as good as resumable cloning, because you have to partition your repo in advance.
Post reply on HN