How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo?
EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM type company in he late '90s/early 2000's before you do that. Please!
How does a sparse checkout not defeat the purpose of a monorepo? I thought monorepos existed so it was easy to make changes that affect the whole codebase and to test those changes. If you only checkout a portion of the files, how are you going to test against the whole repo? EDIT: my overall concern is that it looks like people are reinventing clearcase. Please speak to an older developer who worked at an HP/IBM typ…
Continuous integration tools still check out and test the whole repo. Google has used this approach for over a decade.
Interesting. I've been wanting something like that for submodules. Can the two features be combined? For instance, if you need a single file/directory from another project in your repository.
I thought one of the big benefits of monorepos was that you didn't mess with submodules anymore?
I wish somebody'd write a book on monorepos. I've run into only a handful of their problems when trying to manage production pipelines using just a dozen services, so I'm sure there's tons more (like the purpose behind this command). Nobody mentions the massive investment in time, technical expertise, compute resource, and money required to run large monorepos in production. Also, would emulating this command with a…
Like this one? https://trunkbaseddevelopment.com/
> the core requirement of Continuous Integration that all team members commit to trunk at least once every 24 hours
sparse-checkout, partial-clone, and shallow seem like decent building blocks to make working with very large repos tractable in git. At the same time, the features and their interaction are pretty complicated, so I believe we'll need good "porcelain" abstractions over these building blocks to make the workflow reasonable for average users.
What is your bar for 'average users'? If we're talking project-wide repos rather than entire-org repos, I'd wager the vast majority of projects can use monorepos without special git tooling, and will retain huge productivity benefits vs app/package-per-repo organisation.
Honestly most orgs (with "most" weighted by org, not by headcount) could handle entire-org repos without using any of these features. It's still worth simplifying the workflow and training experience for projects and orgs that grow beyond that, though.
This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
The author of the blog post is on the git team at Microsoft.
At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows) Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree. Works out pretty well. Keep in mind though that sparse checkout still has the e…
Are you sure about that? > This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?
Using sparse-checkout by itself will still download the entire repository and its complete history into .git. If you additionally use the "partial clone" feature, then you can restrict what gets downloaded and stored in .git as well - it will download only the objects that are needed for your selected directories (along with their complete history). On big repositories with long history this might still be too much data, so you might also want to use the "shallow clone" feature (via the --depth flag) to restrict how much history you download.
I have always done —depth=1 for projects I am not a core developer of, but ran into an issue with it being seemingly impossible to do the same with submodules. Golang should have figured this out from day 1 before shipping with a release system built around cloning a repo in its entirety, history and recursive submodules, and all.
Have you tried the --shallow_submodules flag? I haven't used it, but it seems like to should do what you want.
This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
I'm surprised that VFS for Git isn't yet available on GitHub. Surely they are working on adding it? Anyone have an inside scoop?
VFS for Git tool only works on Windows, and is written in C#; it also relies on a fork of Git.