April fool aside, what I really want is a Debian git monorepo of submodules, where each submodule points to the upstream git repository.
What do you do about upstream projects that don't use Git ?
Debian Git Monorepo
101–110 of 157 posts
Re: Debian Git Monorepo
#102Earlier quoted context omitted.
Doesn't google have a lot of internal tooling to make that monorepo actually work for them? The issue isn't so much that large monorepos don't work, it's that you are trading one set of issues for another set of issues. And most popular open source tooling (and I assume all debian-specific tooling) is built around solving the issues of many repositories rather than solving the issues of monorepos.
> Doesn't google have a lot of internal tooling to make that monorepo actually work for them? probably tests coverage and workflow matters: you can work with monorepo if you can check how your change will impact rest of the ecosystem.
At its core, with separate repositories it's generally fine to pretend that any commit effects all code within that repo. So for example you might run all unit tests for every new commit in a PR, or run all deploy scripts when something is committed to master. Dependencies are usually set up one-way, so code pulls it its dependencies (by referencing specific git commits or released packages). So a change in one repository might mean that it pulls in new or updated dependencies, but software depending on you should only see changes once a change is made to its repository.
For a monorepo those assumption don't hold, any given commit likely only affects a tiny fraction of the content of the monorepo. But a change in a library might also immediately radiate outward to anything in that repo that uses the library, so you can't do a naive check based on the directory tree. So you end up with a build system like bazel that can evaluate the entire dependency graph to know what to run. And you have to do that for pretty much all tooling you want to be triggered by code changes.
Add on top of that the scaling issues you have with git as your repository grows
Re: Debian Git Monorepo
#103Earlier quoted context omitted.
This includes the sources of all the Debian packages. In general this is a good idea, and it is close to how nixpkgs works (which is, in some sense, a "distributed monorepo"), but git, the Debian tooling and so on wouldn't actually be able to support a good workflow here.
> it is close to how nixpkgs works Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition. The Nix equivalent to this joke would be storing all of the pac…
I’d say it is close. A purer example would be gittup[1], which has its (very few) upstreams as submodules. But ultimately, Git submodules are just URLs and commit hashes stored in .gitmodules in the repository root, which doesn’t seem all that different from the URLs and hashes of contents passed to fetchgit et al. in Nixpkgs. The capability of being able to patch an arbitrary package locally and have every transitive dependency rebuilt automatically is also the same.
On the other hand, Debian and Fedora maintain full collections of sources used in their releases, while, AFAIU, if you need an old version of something from Nixpkgs, it has fallen out of the official binary cache, and the upstream is gone, you’re on your own. (There have been some efforts to fix that[2], but the first attempt was very limited in scope, and there doesn’t seem to have been a second one.)
Re: Debian Git Monorepo
#104That would be ok if git gave the option to track branches in subfolders and leave other branches alone. I had a look at submodules but I'm not clever enough to understand how it could help me.
Do you mean worktrees? https://git-scm.com/docs/git-worktree
Thanks, this is the pointer I was waiting for!
Re: Debian Git Monorepo
#105I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?
We have 4 interal-only projects that all kind-of depend on each other and are all rolled out at the same time:
backend, frontend and two shell wrappers for the frontend (electron for windows, cordova for ipad)
The total code base is still small enough for network/file system to not care (~10 devs working 10 years.)
Since we're on Azure Devops, we currently have to create a pull requests for every project. If you change the naming of an API variable, you'll have to create two pull requests. You'll have to review those individually. The automatically triggered test runs will fail against the respective "old" version of each other and create noise.
If, instead, we'd have a structure like this, the changes could be together: /src - backend (C#) - frontend (TypeScript) - ...
Note that we're not planning to have a "shared" library or "common" editorconfig files, or changing _anything_ about code internals. Just tracking the folders in git together.
Re: Debian Git Monorepo
#106Earlier quoted context omitted.
from what I've heard, git become much friendlier with monorepos in recent years because of microsoft involvement. Windows repo, pretty sizeable, is now using git, and it supposedly works even without strange things like vfs for git.
It's not only Windows that uses Git at Microsoft, but Sharepoint and Office (which includes the on-prem version of SharePoint). In terms of repo size Windows and Office are similar. I was part of the team that migrated Sharepoint from a Perforce clone to Git and helped build the tooling to allow Office to move as well. VFS for Git [1] and Scalar [2] are really good pieces of software. [1] - https://github.com/microso…
I hope one of these "large file/many files" extensions becomes part of git core eventually. They are too much of a hassle to deal with and many servers don't even support them.
Re: Debian Git Monorepo
#107Earlier quoted context omitted.
I struggle to understand the upside of avoiding monorepos. A lot of people think it means you can't decouple releases, modularize builds and similar with a monorepo, but that's simply not true. The efficiency at which you can refactor and upgrade your codebase cannot be understated.
every monorepo I've seen suffers from a shit CI solution. Other teams break your builds. Builds get longer and more complex. Feedback loops slow down. Engineering velocity slows down. It works at Google because they spent, literally, hundreds of millions of dollars getting it working.
It's not even clear whether a monorepo stretches across teams or not. I think many people are argueing about different things.
Re: Debian Git Monorepo
#108Earlier quoted context omitted.
Google's monorepo includes snapshots of everything upstream that Google's proprietary code depends on - starting from the kernel and glibc and core language toolchains, and all the way up to various open-source libraries and modules in c/python/java/whatever.
But 3p sources likely are for specific stable version? Linus can't break Google by submitting bug into linux kernel?
Yes.
Re: Debian Git Monorepo
#109Earlier quoted context omitted.
I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. It really depends on how that's set up. And how is sparse checkout worse for discoverability? With multiple repos it's even harder to find what you want sometimes if you are talking about 100's of random repos that aren't organ…
> I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. In abstract I agree. However when I'm trying to get my code working having test failures in code that isn't even related to the problem I'm working on is annoying and I can't switch tasks to work on this new failure when the…
How could broken code (or broken tests) be merged in master ? That is a rhetorical question, of course it happends, and of course this is the root issue you would be facing
Re: Debian Git Monorepo
#110Putting aside that this is an April Fool's joke, I like the last part:
> This time, I’m cruel to Git: can it handle a repository of this size? In 2009 it could not. In 2024 it can.