Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git partial clone lets you fetch only the large file you need

about.gitlab.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Git partial clone lets you fetch only the large file you need

#41

Post by w0m »

jfkebwjsbx wrote: post #27
vvanders wrote: post #19

Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.

P4 was great for its time if you could pay for it, but it is definitely not a competitor anymore. Git LFS has been just fine for multiterabyte repositories for years.

p4 is still great, that there are workarounds to be usable on similar workloads on git doesn't takeaway p4s inherent advantages. they're two very different tools.

Re: Git partial clone lets you fetch only the large file you need

#42

Post by jayd16 »

vvanders wrote: post #19

Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.

P4 is fine on paper. I just wish the the client didn't crash and the server didn't lock up as often as it does. P4V is a mess.

Re: Git partial clone lets you fetch only the large file you need

#43

Post by vvanders »

w0m wrote: post #41
jfkebwjsbx wrote: post #27

Earlier quoted context omitted.

P4 was great for its time if you could pay for it, but it is definitely not a competitor anymore. Git LFS has been just fine for multiterabyte repositories for years.

p4 is still great, that there are workarounds to be usable on similar workloads on git doesn't takeaway p4s inherent advantages. they're two very different tools.

Thank you, that phrases it much better than I could. They are very different tools(although I hear good stuff about P4 Fusion).

Now if someone wanted to make an open source P4 replacement that would be a neat thing.

Re: Git partial clone lets you fetch only the large file you need

#44

Post by dagmx »

vvanders wrote: post #40
jasondclinton wrote: post #21

Earlier quoted context omitted.

This kind of comment isn't helpful. Of course, there have been ways to copy large files around since there were networks. What's new in this protocol enhancement is that this works within the context of a Merkle tree-based technology (upon which all DVCS's are based). To use your analogy, yes this is a wheel but it's built with rubber instead of wood and iron.

I guess I should have expanded more. DVCS is in direct opposition of workflows that include binary files(yes I'm aware that git lfs has locking, it's also centrally orchestrated) because you can't merge almost every binary format. We were using P4 ~15 years ago for these workflows and rather than understanding what made them work people are just rediscovering the same problems that have already been solved. My guess…

I think this is a very p4 centric view of the world.

Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching?

Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there.

The issue is neither is a silver bullet for the others workflow and needs, and they both suck horribly for mixed code and binary asset workflows.

That's not even considering cost.

Meanwhile film studios generally prefer keeping the considerations separate and using symlinks or URI to their data store and that works really well. But that doesn't work great for remote workflows.

So again, I think you're applying a very p4, game centric view to this. There are lots of different use cases and team structures that none of these version control systems are able to address in their entirety.

Re: Git partial clone lets you fetch only the large file you need

#45

Post by mikepurvis »

madsbuch wrote: post #39
mikepurvis wrote: post #14

Earlier quoted context omitted.

Do you actually need clones in that scenario? I worked on a build system that grabbed source from several hundred repos at the starting point, and it turned out to be way faster to just grab it all as tarballs with aria2c.

Grapping the tarbell from where? To my best knowledge, tarbell export is not a part of git, but something git hosts provide. Git is a distributed VCS, and we should support keeping it that way.

Almost any project you work on will have an authoritative copy of the repo in some kind of web-accessible tool, most of which provide a tarball-download function.

And GitHub's scheme is pretty much a de-facto standard at this point—GitLab's implementation is an exact copy of it, for example:

    https:////archive/.tar.gz
Edit to add: Also, git-archive --remote is actually most of the way there, but it's not an HTTP download, of course. :(

Re: Git partial clone lets you fetch only the large file you need

#46

Post by vvanders »

dagmx wrote: post #44
vvanders wrote: post #40

Earlier quoted context omitted.

I guess I should have expanded more. DVCS is in direct opposition of workflows that include binary files(yes I'm aware that git lfs has locking, it's also centrally orchestrated) because you can't merge almost every binary format. We were using P4 ~15 years ago for these workflows and rather than understanding what made them work people are just rediscovering the same problems that have already been solved. My guess…

I think this is a very p4 centric view of the world. Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there. The issue is neither is a silver bullet for the others workflow and needs, and they both su…

If by P4 centric you also mean SVN as well then sure.

I will say however that if you think locking is optional then you already don't understand these workflows and why they're so critical. Art/design/animation doesn't care that "they should not have been touching the file" they just care that they have to throw away two days of work because someone made multiple edits to the same package file. I've literally seen multiple teams almost come to blows when this happens.

You can separate code from art/assets. That comes at an integration and iteration cost, it'll drive your designers mad.

There are workflows out there where code is not the first class citizen, in those cases I've seen git shoehorned in and untold pain follows.

Re: Git partial clone lets you fetch only the large file you need

#47

Post by Keverw »

danbolt wrote: post #24

In the AAA games industry git has been a bit slower on the uptake (although that’s changing quickly) as large warehouses of data are often required (eg: version history of video files, 3D audio, music, etc.). It’s nice to see git have more options for this sort of thing.

Surprised this new idea doesn’t support object storage. Sounds like Git LFS would still be the right way to go for repos with assets for games like meshes, sounds, etc.

However I’ve heard many studios use Perforce instead. However not being open source is a downside to some, but I don’t really know too much about it personally.

Then if working with a lot of non code files, sounds like some solutions have locking. I guess not two people could edit the same Blender or PSD file at the same time and then merge them later on.

Kinda wouldn’t surprise me if some companies actually run multiple versioning control systems. Code on one system, game assets on another.

Re: Git partial clone lets you fetch only the large file you need

#48

Post by Ididntdothis »

derefr wrote: post #7

Has anyone used Git submodules to isolate large binary assets into their own repos? Seems like the obvious solution to me. You already get fine-grained control over which submodules you initialize. And, unlike Git LFS, it might be something you’re already using for other reasons.

I have tried sub modules but it’s way too easy to shoot yourself in the foot. Not very sustainable in a team with different levels of git knowledge.

Re: Git partial clone lets you fetch only the large file you need

#49

Post by 01100011 »

vvanders wrote: post #19

Also known as workspace views in P4. It's interesting to see the wheel reinvented. We used to run a 500gb art sync/200gb code sync with ~2tb back end repo back when I was in gamedev. P4 also has proper locking, it is really the is right tool if you've got large assets that need to be coordinated and versioned. Only downside of course is that it isn't free.

I love P4 for just working, but I absolutely can't stand the limited shelving ability. I ended up writing a helper program that lets me shuffle local changes off to a git repo just so I could manage working on several overlapping changelists. Perforce would be so much more usable if they would include this sort of basic functionality right out of the box. The thing git gets right is that you often need to juggle several threads of change at the same time, and those threads may have complex branching as you try out different approaches and combine the best pieces at the end.

Re: Git partial clone lets you fetch only the large file you need

#50

Post by maccard »

dagmx wrote: post #44
vvanders wrote: post #40

Earlier quoted context omitted.

I guess I should have expanded more. DVCS is in direct opposition of workflows that include binary files(yes I'm aware that git lfs has locking, it's also centrally orchestrated) because you can't merge almost every binary format. We were using P4 ~15 years ago for these workflows and rather than understanding what made them work people are just rediscovering the same problems that have already been solved. My guess…

I think this is a very p4 centric view of the world. Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there. The issue is neither is a silver bullet for the others workflow and needs, and they both su…

> but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching?

Because there's 300+ people working on a project, and it's not feasible to know what every other person is working on or planning on working on.

The file lock (code can be merged too, just like git, so this is only really for binary assets) is a crude communication tool saying "hey I'm using this file".

> Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there.

I think you're applying a git biased view here. I don't think perforce is unsuitable for code heavy projects at all (see workspace views as a prime example), and for the majority of people, having a central server isn't an issue. Most people treat github/gitlab as a centralised server anyway. I've _never_ in a decade of programming heard someone suggest adding an extra remote to git so I can share your changes, it's always been "push it as a separate branch and I'll merge it". If you have a missing internet connection, you're likely not able to share code _anyway_, and with p4 you can always reconcile offline work when you're back online.

Post reply on HN