Live data from Hacker News

Bring your monorepo down to size with sparse-checkout

github.blog

11–20 of 68 posts

Re: Bring your monorepo down to size with sparse-checkout

#11
post #5

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

> For instance, if you need a single file/directory from another project in your repository.

The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.

Re: Bring your monorepo down to size with sparse-checkout

#12
I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production.

Also, would emulating this command with a repo of submodules not work?

Re: Bring your monorepo down to size with sparse-checkout

#13
post #11
post #5

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

> For instance, if you need a single file/directory from another project in your repository. The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.

Sometimes you don't own the other repo.

Re: Bring your monorepo down to size with sparse-checkout

#14
post #5

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows)

Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree.

Works out pretty well.

Keep in mind though that sparse checkout still has the entire repository loaded in the `.git` object store, it just doesn't expose it in the worktree.

Re: Bring your monorepo down to size with sparse-checkout

#15
post #3

I have to tell it which directories I want? That seems like work the tool could do. Also, the granularity should be at the file level, not directory.

It's based on glob paths. I think you can specify it at whatever level you want. Also you can use wildcards.

Re: Bring your monorepo down to size with sparse-checkout

#16

sparse-checkout, partial-clone, and shallow seem like decent building blocks to make working with very large repos tractable in git. At the same time, the features and their interaction are pretty complicated, so I believe we'll need good "porcelain" abstractions over these building blocks to make the workflow reasonable for average users.

What is your bar for 'average users'?

If we're talking project-wide repos rather than entire-org repos, I'd wager the vast majority of projects can use monorepos without special git tooling, and will retain huge productivity benefits vs app/package-per-repo organisation.

Re: Bring your monorepo down to size with sparse-checkout

#17
This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3]

[1]: https://github.com/microsoft/VFSForGit

[2]: https://news.ycombinator.com/item?id=14411126

[3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

Re: Bring your monorepo down to size with sparse-checkout

#18
post #14
post #5

Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the e…

Are you sure about that?

> This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory

If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

Re: Bring your monorepo down to size with sparse-checkout

#19
post #14

Earlier quoted context omitted.

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the e…

Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree.

Edit:

   wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue"
   twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):
   twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):
   twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):


   wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/
   install_linux_gcc54
Post reply on HN