Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debian Git Monorepo

blog.liw.fi

Post reply on HN
Print view View original

151–157 of 157 posts

Re: Debian Git Monorepo

#151

Post by BlackFly »

JackSlateur wrote: post #109
bluGill wrote: post #75

Earlier quoted context omitted.

> I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. In abstract I agree. However when I'm trying to get my code working having test failures in code that isn't even related to the problem I'm working on is annoying and I can't switch tasks to work on this new failure when the…

How could broken code (or broken tests) be merged in master ? That is a rhetorical question, of course it happends, and of course this is the root issue you would be facing

There are multiple ways that code gets merged into master and ends up broken.

First, the one where everyone does everything correctly: CI executions do not run serially because when too many people are producing a lot of code, you need them to run at the same time. So you have two merge requests which are done around the same time A and B, they each see a commit C before each other. Say merge request A deletes a function or class or whatever that merge request B uses. Of course merge request A deleted all uses of that function but could not delete the use by B since it was not seen. A + C passes all CI checks and merges. B + C passes all CI checks and merges. A + B + C won't compile since B is using a function deleted by A. If you are lucky, they touch the same files and B doesn't merge due to a merge conflict and the rebase picks it up, otherwise broken master.

Then you will typically have emergency commits to hotfix issues which might break other things.

Then you will have hidden runtime dependencies that won't trigger retests before merge due to being hidden, but every subsequent change to that repo will fail.

Then you will have certificates, dependencies on external systems that go away.

Re: Debian Git Monorepo

#152

Post by BlackFly »

hobofan wrote: post #90
BlackFly wrote: post #31

Earlier quoted context omitted.

The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on tha…

Things become difficult at scale regardless of mono- or multirepo. You also have to build dedicated tooling if you heavily lean into splitting things into a lot of repositories, in order to align and propagate changes throughout them.

Sure, but polyrepos don't break with scale in the same way as monorepos. You only need additional tooling when you are trying to coordinate homogeneity a scale larger than your manual capability. Autonomous services don't typically do that kind of coupling without cohesion that people naturally find necessary in a monorepo and you can build cooperative and coexisting products without that kind of coupling.

When I read the white papers by google or uber on their monorepos, when I see what my company is building, it is just a custom VCS. Everything that was thrown away initially gets rebuilt over time. A way to identify subprojects/subrepositories. A way to check out or index a limited number of subprojects/subrepositories. A way to define ownership over that subproject/subrepository. A way for that subproject/subrepository to define its own CI. A way to only build and deploy a subproject/subrepository. Custom build systems. Custom IDEs.

The entirety of code on the planet is a polyrepo and we don't have problems dealing with that scale like we would have if we stuffed it all in one repo like this debian monorepo shows. Independence of lifecycle is important, and as a monorepo scales up people rediscover that importance bit by bit.

Re: Debian Git Monorepo

#153

Post by jorvi »

zer00eyz wrote: post #149
jorvi wrote: post #141

Earlier quoted context omitted.

Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. People see a bit of code they want to improve, they improve it, and then instead of just opening a PR (and perhaps iterating on it), now they have to learn a weird ancient e-mail workflow and will probably get chewed out for messing it up. No one is saying to take away mail-in patches but it is positively arc…

>> Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. Is it friction? Or is it a filter? You might remember being a kid and there was the sign in front of the ride that said "you must be at least this high to ride".... The kernel dev process isnt for casuals. It's designed that way. There's a lot of folks out there who have popular projects on GitHub who are…

> Is it friction? Or is it a filter?

Friction.

> There's a lot of folks out there who have popular projects on GitHub who are over the endless stream of BS from AI generated pull requests.

So be stringent. First below-par PR get some guidance, pointers and perhaps a reprimand. Second time, a warning, third time a ban.

> You should really dig in deep to what goes on with the kernel, the work flow, why it is that way and why GitHub is outright incapable of supporting kernel dev (there are reasons)....

Code is code. If someone has an improvement, they can offer their new code.

I’m sure the kernel has a unique workflow, but it ultimately boils down to that, no?

Re: Debian Git Monorepo

#154

Post by mewpmewp2 »

Archelaos wrote: post #96
mewpmewp2 wrote: post #59

Earlier quoted context omitted.

The argument then is also that monorepo allows you to use this as "shared modules" instead of published "libraries". The point is you have some code that you want to reuse, and either within a monorepo or between multiple repositories.

The question is, when it is wise to do so. Changing the public interface of a "shared module" or published "library" comes with a liability. Code reuse is almost a no brainer in the case of a standard library that rarely or never changes. Using a permanently changing modul whose modifications provide no real benefits for your project makes no economic sense. The sweetspot is somewhere in between. What the parent actu…

With monorepo he wouldn't have had to worry about this decision, since the decision to go back and forth would have been much lower cost. It's low cost decision either way and low cost impact if you make a wrong decision.

Re: Debian Git Monorepo

#155

Post by richardwhiuk »

eichin wrote: post #135
richardwhiuk wrote: post #60

Earlier quoted context omitted.

Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.

it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)

I meant what upstream Debian includes.

Re: Debian Git Monorepo

#156

Post by dTal »

It's worth noting that Void Linux (and maybe others) does indeed keep all its (largely declarative) package recipes and build tooling in a single git monorepo, although it does not keep copies of the upstream source because (package, version, url, checksum) is assumed to be idempotent, and some of the stated advantages are real:

>Simpler collaboration: every package uses the same process, and the same tools, and it’ll be easier than ever to help with other people’s packages.

>Enables distribution-wide changes in general: With all the source code for everything in one tree, in on repository, it’s feasible to make changes to Debian that affect many packages. For example, back in the day Debian took seven years to migrate /usr/doc to /usr/share/doc, and that can now be done in one commit.

Re: Debian Git Monorepo

#157

Post by ghosty141 »

BlackFly wrote: post #150
ghosty141 wrote: post #80

Earlier quoted context omitted.

> Similarly, once you are too popular a library in a monorepo, you will never do any atomic breaking API changes since it would require updating too many downstreams. This happens no matter which repo type. Even worse if a project chooses to update after a while, its far more painful having to do the changes after losing the context you had when you did the original changes. If you want a monorepo, libraries being on…

This doesn't happen in a poly repo because you can just do it. You release version 2.0.0 of something and downstreams update at their own pace. Diverging as you call it. But this isn't a problem. If 1.0.0 is a finished product then why do you ever need to move to 2.0.0 if you don't need the new features? The issue in the monorepo is that if you are too popular the change must happen all at once or with copying (fake…

> But this isn't a problem. If 1.0.0 is a finished product then why do you ever need to move to 2.0.0 if you don't need the new features?

Fair point, I assume (from personal experience at the places I've worked at) that updating the library is inevitable and doing so at a later date tends to be more painful than doing these migrations all at one.

> It isn't a feature of the monorepo because you can still diverge by copying, forking or merely stopping support for the old library and this becomes more and more necessary at scale

This is a problem if all the projects in the monorepo are not actually related. But imagine if all these subprojects are bundled as one OS image, in that case it is very rare that you want multiple library version.

At very large scale I can see your point, I don't have experience there so I can't really argue.

Post reply on HN