Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bring your monorepo down to size with sparse-checkout

github.blog

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Bring your monorepo down to size with sparse-checkout

#21

Post by gdxhyrd »

0xbadcafebee wrote: post #12

I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a…

What problems did you encounter with just a few services?

Monorepos should be straightforward unless you are managing the code of >1k engineers.

Re: Bring your monorepo down to size with sparse-checkout

#23

Post by stolee »

vlovich123 wrote: post #18
war1025 wrote: post #14

Earlier quoted context omitted.

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the e…

Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand.

Git’s partial clone is a more natural way of achieving the same outcome.

Re: Bring your monorepo down to size with sparse-checkout

#24

Post by xyzzy_plugh »

ComputerGuru wrote: post #8

I have always done —depth=1 for projects I am not a core developer of, but ran into an issue with it being seemingly impossible to do the same with submodules. Golang should have figured this out from day 1 before shipping with a release system built around cloning a repo in its entirety, history and recursive submodules, and all.

FWIW Go modules address this pretty well.

Re: Bring your monorepo down to size with sparse-checkout

#25

Post by hackernews3000 »

0xbadcafebee wrote: post #12

I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a…

Like this one? https://trunkbaseddevelopment.com/

Re: Bring your monorepo down to size with sparse-checkout

#26

Post by wikibob »

divbzero wrote: post #17

This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

I'm surprised that VFS for Git isn't yet available on GitHub. Surely they are working on adding it? Anyone have an inside scoop?

Re: Bring your monorepo down to size with sparse-checkout

#27

Post by WorldMaker »

divbzero wrote: post #17

This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

This sounds like a parallel effort (along with the commit graph work) to keep pushing that better. The article's writer (who also wrote most of the blog posts on the commit graph work) mentions a "three million file repository" used for testing in this article and that would of course sound like the Windows repo.

It's also I'd imagine not mutually exclusive effort. It seems like exactly like something you would want in combination with something like VFS at scale, as it reduces the number of materialized versus virtual objects in both the git working copy and the git object database. If you've got millions or billions of objects and files, even reducing the number of virtual placeholders I would imagine is probably a big win.

Re: Bring your monorepo down to size with sparse-checkout

#28

Post by WorldMaker »

stolee wrote: post #6
tantalor wrote: post #3

I have to tell it which directories I want? That seems like work the tool could do. Also, the granularity should be at the file level, not directory.

The sparse-checkout patterns match at the file level, so you can always use that (without “cone mode”) if you want. It becomes difficult to match an exact file list as people add files to projects: you require every other user to update their patterns to match the newly-added file.

Just keep in mind as the article points out that without "cone mode" is potentially a lot slower, and that's why cone mode exists.

Re: Bring your monorepo down to size with sparse-checkout

#29

Post by kyrra »

wikibob wrote: post #26
divbzero wrote: post #17

This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

I'm surprised that VFS for Git isn't yet available on GitHub. Surely they are working on adding it? Anyone have an inside scoop?

2 reasons: VFS is a fork of normal Git. There is no Linux client.

Also remember that there are many Git clients that work with normal Git repos. Like libgit and others. I doubt you'll see wide spread support for it unless MS can upstream it into the main Git implementation, and maybe some the the primary libraries.

This is one nice argument for Mercurial, where there is only a single implementation, so adding big new changes can be easier.

Re: Bring your monorepo down to size with sparse-checkout

#30

Post by scarejunba »

monocasa wrote: post #13
mappu wrote: post #11

Earlier quoted context omitted.

> For instance, if you need a single file/directory from another project in your repository. The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.

Sometimes you don't own the other repo.

Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.
Post reply on HN