Git partial clone lets you fetch only the large file you need • Hacker News Board

Re: Git partial clone lets you fetch only the large file you need

#61

Post by beagle3 » Sat, Mar 14, 2020, 9:47 PM UTC

pas wrote: post #58

beagle3 wrote: post #51

There is one note piece to the puzzle to make git perfect for every use case I can think of: store large files as a list of blobs broken down by some rolling hash a-la rsync/borg/bup. That would e.g. make it reasonable to check in virtual machine images or iso images into a repository. Extra storage (and by extension, network bandwidth) would be proportional to change size. git has delta compression for text as an op…

Do ISOs and other large blob types support only partial (block) modification? Wouldn't all subsequent blocks change too?

Sometimes they do - e.g. if you replace a file in the ISO that is the same size up to block alignment, which is common when e.g. editing a text file or recompiling an executable with a minor change. They almost always do when it's a VM image representing a disk - only some blocks change every write.

However, with self synchronizing hashes of the kind used by rsync bup and borg, it doesn't matter - you could have a 1TB file, delete a single byte at position 100 - and you only need to store or transfer one new block (with average size 8KB for rsync, configurable for borg) if you already have a copy of the version before the change.

It's somewhat comparable with diff/patch but not exactly; it's worse in that change granularity is only specified on average; It's better in that it works well on binary files, does not require a specific reference diff (can reference all previous history), and efficiently supports reordering as well small changes - if you divide a 4000 line text file to four 1000-line sections and reorder them 1,2,3,4 -> 3,1,4,2 you will find the diff/patch to be as long as a new copy, whereas a self synchronizing hash decomposition will hardly take any space for the reordered file given the original.

Re: Git partial clone lets you fetch only the large file you need

#62

Post by andrewshadura » Sat, Mar 14, 2020, 10:10 PM UTC

itroot wrote: post #8

Also --reference (or --shared) is a good parameter to speed-up cloning (for build, for example), if you have your repository cached in some other place. I was using it a long time ago when I was working on system that required to clone 20-40 repos to build. This approach decreased clone timings by an order of magnitude.

Careful, with extra large repositories it actually slows down the cloning while, obviously, significantly reducing the space usage.

Re: Git partial clone lets you fetch only the large file you need

#63

Post by dagmx » Sat, Mar 14, 2020, 10:13 PM UTC

vvanders wrote: post #56

dagmx wrote: post #54

Earlier quoted context omitted.

I've worked as a pipeline supervisor at one of the biggest VFX studios. I very much understand these workflows. I've worked as an artist and a tech artist in perforce and SVN workflows too. I've had to support the workflows of a 1000+ workforce across multiple locations. I don't think attributing my disagreement with you to not understanding workflows is a fair characterization. I still think locking , while useful,…

I think we'll just have to agree to disagree. It sounds like we just come from different development cultures. Your solution to lack of locking sounds like a top-down hierarchy that wouldn't be flexible enough to support the teams I've worked with. Having seen both approaches(and how they break down) I'll take a centralized locking solution over communication mistakes that lead to days of work being lost.

That's fair to agree to disagree but I also again think it's unfair to characterize my pipelines as not flexible enough.

For context I developed the publishing pipelines for the majority of departments in the studio. I have several hundreds of assets being published through my pipelines on a daily basis, if not more, from a variety of departments.

We only hit collisions on a very rare basis, and which were often resolved in an hour or two in the worst case.

We've scaled this from very small teams to large ones, from very scrappy realtime productions to feature length offline rendered films.

I don't doubt that locking helps. I just argue that maybe it's not as critical as people make it out to seem.

Re: Git partial clone lets you fetch only the large file you need

#64

Post by microtherion » Sat, Mar 14, 2020, 10:57 PM UTC

That seems quite useful, though Git LFS mostly does the job.

One of my biggest remaining pain points is resumable clone/fetch. I find it near impossible to clone large repos (or fetch if there were lots of new commits) over a slow, unstable link, so almost always I end up cloning a copy to a machine closer to the repo, and rsyncing it over to my machine.

Re: Git partial clone lets you fetch only the large file you need

#65

Post by danbolt » Sat, Mar 14, 2020, 11:19 PM UTC

jfkebwjsbx wrote: post #25

danbolt wrote: post #24

In the AAA games industry git has been a bit slower on the uptake (although that’s changing quickly) as large warehouses of data are often required (eg: version history of video files, 3D audio, music, etc.). It’s nice to see git have more options for this sort of thing.

Git LFS has been a thing for years, though.

You’re absolutely right, but larger developers and publishers have been slower to adopt.

P4’s GUI/model is also intuitive for non-programming roles to learn and use historically compared to git, so a team with wide skills can ramp up quickly with a unified toolset. A less-technical manager gets a GUI that has versioning across changes from a multidisciplinary team. You can probably guess what inertia that has in a space with higher turnover compared to other industries.

As mentioned, things are changing though. git and GitHub have become a mainstay and are what new programmers likely learn in schools. This has a trickle effect on new projects with smaller teams and results in more investment into git setups. I use git in a AAA context at work, and it’s not uncommon to find sentiments from more seasoned game programmers on git that are similar to HN comments about the latest fad in web frameworks.

Re: Git partial clone lets you fetch only the large file you need

#66

Post by adrianmonk » Sat, Mar 14, 2020, 11:20 PM UTC

pas wrote: post #58

beagle3 wrote: post #51

There is one note piece to the puzzle to make git perfect for every use case I can think of: store large files as a list of blobs broken down by some rolling hash a-la rsync/borg/bup. That would e.g. make it reasonable to check in virtual machine images or iso images into a repository. Extra storage (and by extension, network bandwidth) would be proportional to change size. git has delta compression for text as an op…

Do ISOs and other large blob types support only partial (block) modification? Wouldn't all subsequent blocks change too?

It really depends on the type of file. ("Other large blob types" is a rather broad category.)

One obvious example where you could have a lot of common blocks (even following the offset where a change was made) is zip files. The zip format basically compresses each file individually and then concatenates all that together.

Let's say you have a build and it packages the results up as a big zip file. (Java builds often do this. A jar is a special type of zip file.) If you change a few source files and rebuild, and if your build is deterministic (and/or incremental), then the new zip file will contain a lot of the same stuff as the previous version. And if your zip archiver is deterministic (pretty safe assumption), it should produce a zip file that is mostly the same sequences of bytes as the previous zip file, even if there are changed files in the middle.

If you write a .tar.gz archive, then one change in the middle will throw everything off from that point on because it compresses the whole archive instead of individual files. In theory a binary diff can work around this by first undoing the gzip that was done to create each large blobs, then doing a binary diff on that, and then arranging to be able to recreate what gzip did. Obviously that's messy.

Of course, not every file is an archive. Some are filesystems. But any writable filesystem (notably not including ISOs) that is capable of being used on a hard disk will of necessity not rewrite everything. If it did, changing on one file on a filesystem would take hours because the rest of the partition would have to be rewritten.

Another obvious type of big blob is multimedia. I don't know a lot of specifics, but I would think file formats meant for editors would keep changes localized for reducing IO (for example, so that changes in a non-linear video editor don't need to write a giant file), but formats meant for export and delivery might change the whole file since they're aiming for small size.

Re: Git partial clone lets you fetch only the large file you need

#67

Post by vvanders » Sat, Mar 14, 2020, 11:24 PM UTC

dagmx wrote: post #63

vvanders wrote: post #56

Earlier quoted context omitted.

I think we'll just have to agree to disagree. It sounds like we just come from different development cultures. Your solution to lack of locking sounds like a top-down hierarchy that wouldn't be flexible enough to support the teams I've worked with. Having seen both approaches(and how they break down) I'll take a centralized locking solution over communication mistakes that lead to days of work being lost.

That's fair to agree to disagree but I also again think it's unfair to characterize my pipelines as not flexible enough. For context I developed the publishing pipelines for the majority of departments in the studio. I have several hundreds of assets being published through my pipelines on a daily basis, if not more, from a variety of departments. We only hit collisions on a very rare basis, and which were often reso…

What happens if you're not around to drive the process? What about if you don't have the organizational backing to drive the process? What if a team goes AWOL or isn't bought in to your process? I've seen variants of all those happen in production in one form or another.

At the end of the day humans make mistakes, especially when involving communication. I'd rather have a physical system that prevents breaks instead of requiring cross-team/cross-discipline coordination.

Maybe gamedev is much more coupled than film(we regularly had design, animation, art and code touching the same common core packages). Look at Unreal or any other gamedev pipeline and you'll see a bias for locking source control solutions.

Re: Git partial clone lets you fetch only the large file you need

#68

Post by danbolt » Sat, Mar 14, 2020, 11:30 PM UTC

Keverw wrote: post #47

danbolt wrote: post #24

In the AAA games industry git has been a bit slower on the uptake (although that’s changing quickly) as large warehouses of data are often required (eg: version history of video files, 3D audio, music, etc.). It’s nice to see git have more options for this sort of thing.

Surprised this new idea doesn’t support object storage. Sounds like Git LFS would still be the right way to go for repos with assets for games like meshes, sounds, etc. However I’ve heard many studios use Perforce instead. However not being open source is a downside to some, but I don’t really know too much about it personally. Then if working with a lot of non code files, sounds like some solutions have locking. I g…

I think in terms of game production, software licensing usually isn’t the largest cost center for a project. Proprietary software isn’t a concern as much, given that games traditionally are “shipped” and then completed. (Note that this changes as games that are more online service-based with live operations, rather than a specific release date and a “final” copy sent for production; the internet has changed things a lot)

You’re more right than you think about multiple versioning systems, although keeping synchronized becomes an issue. Perforce is a bit of a boon for management, as they get a GUI for versioning across a multidisciplinary team.

Re: Git partial clone lets you fetch only the large file you need

#69

Post by adrianmonk » Sat, Mar 14, 2020, 11:36 PM UTC

dagmx wrote: post #44

vvanders wrote: post #40

Earlier quoted context omitted.

I guess I should have expanded more. DVCS is in direct opposition of workflows that include binary files(yes I'm aware that git lfs has locking, it's also centrally orchestrated) because you can't merge almost every binary format. We were using P4 ~15 years ago for these workflows and rather than understanding what made them work people are just rediscovering the same problems that have already been solved. My guess…

I think this is a very p4 centric view of the world. Locking helps with preventing collisions, but honestly the issue is still always communication. Why are people even touching files they shouldn't be touching? Meanwhile perforce is a pain for code heavy projects and requiring a central perforce server. Git works great there. The issue is neither is a silver bullet for the others workflow and needs, and they both su…

> Why are people even touching files they shouldn't be touching?

I don't think it's accurate to say they shouldn't be touching the files.

It's possible to make two unrelated changes in the same binary file just as it is possible to make two unrelated changes in a source file (or other merge-friendly text file).

Just as there may be nothing wrong if one person changes the function foo() in a file and someone else changes the function bar() in that same file, there may be nothing wrong if one person opens a CAD file and makes a change to a drawing in one part and another person makes a change to a different part of the same drawing.

In that case, they could coordinate by communicating (even though their tasks are unrelated) but then they're just doing the same thing as locking the files but manually and informally (and probably inconsistently) without the benefits of automation.

Of course there are times when locking catches failures to communicate, but that doesn't mean that that's what locking is for.

Re: Git partial clone lets you fetch only the large file you need

#70

Post by dagmx » Sat, Mar 14, 2020, 11:41 PM UTC

vvanders wrote: post #67

dagmx wrote: post #63

Earlier quoted context omitted.

That's fair to agree to disagree but I also again think it's unfair to characterize my pipelines as not flexible enough. For context I developed the publishing pipelines for the majority of departments in the studio. I have several hundreds of assets being published through my pipelines on a daily basis, if not more, from a variety of departments. We only hit collisions on a very rare basis, and which were often reso…

What happens if you're not around to drive the process? What about if you don't have the organizational backing to drive the process? What if a team goes AWOL or isn't bought in to your process? I've seen variants of all those happen in production in one form or another. At the end of the day humans make mistakes, especially when involving communication. I'd rather have a physical system that prevents breaks instead…

It's very rare that someone needs to be around to oversee the process. Tooling guides the vast majority of users in to a workflow that works while still being flexible should they need it.

Lack of organizational backup, do you mean cultural from the studio or infrastructure? Both are a problem no matter what solution you pick.

If a team goes AWOL, that's on them. The tooling usually allows for some amount of arbitrary workflow but they can't go completely off the rails. But that's true of p4 too. So I think that scenario would have to be more specific.

And yes people make mistakes, and you need tooling to guide them. Locking is a tool, but it's not the only tool. I feel very much that many workflows use it to hide deeper issues. That's not to say it's not valid, it is, but it's not a panacea either.

Unreal heavily favors perforce and SVN because that's what it was designed around. There's no absolute reason it could not work with other versioning systems and their paradigms if it came to being necessary.

Unity on the other hand is quite happy to work with any version control system, and works quite well with git or perforce.

You again seem to be trying to approach this from the angle of only the system you're familiar with working. But maybe try stepping outside the box and seeing if your workflow isn't a byproduct of your tools.

After all, you were asking git users to look at perforce as the solution. I don't think it's fair to then go ahead and assume that p4 is the only workable solution.