Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debian Git Monorepo

blog.liw.fi

Post reply on HN
Print view View original

91–100 of 157 posts

Re: Debian Git Monorepo

#91

Post by sethammons »

gtirloni wrote: post #88
sethammons wrote: post #74

Earlier quoted context omitted.

that tells me you prematurely split something. I worked at a high scale, high available system with multiple dozens of teams, each with several repos. If we had to sync changes across repos, we joined the repos because they were, well, joined. In the early days, we had shared libraries and keeping those in sync was a pita. Then we realized we should not be doing what we were doing. The flow should be: (1) update a li…

> (2) code that wants/needs the new functionality updates the usage at its leisure. That's how you end up with infinite support costs in a large org. I think the point is that these changes are done atomically.

that is the theory. The reality is that your team can work on fixing an issue now, but the consuming team might be able to get to it next quarter due to competing priorities. Getting multiple teams to agree to work together at the same time is much, much harder than allowing teams to update on their own schedule. You do eventually have to press the organization and deprecate the older thing.

The alternative is you do the work for them - which you may or may not have the expertise for, and the other team may not even have time for the needed code review.

Re: Debian Git Monorepo

#92

Post by skydhash »

jorvi wrote: post #52
xprn wrote: post #15

Earlier quoted context omitted.

I would actually think that monorepos suck with Github and others like that. Looking at the Linux kernel for example, that seems to me like a well executed monorepo outside of Github. From what I’ve heard and read, they specifically use Git and mailing lists instead of Github or others because Github wouldn’t work for what Linux is (aside from as a mirror). There might very well be some context I’m missing, but that’…

Github / Gitlab would work great and reduce friction for kernel contribution. The refusal to switch or provide the alternate path is mostly inertia.

Sometimes friction is necessary for quality. More often than not, I’ve seen github repos where issue is treated as a project manager and there are lots of drive-by PRs.

Re: Debian Git Monorepo

#93

Post by ithkuil »

IshKebab wrote: post #53
ithkuil wrote: post #19

Earlier quoted context omitted.

The opensource tooling to handle large monorepos is just not there. The monorepo at Google with the massively distributed build system and all other tooling make it very effective to work with. You cannot just take one part of that system and expect to magically reap the benefits. The main benefit I see in practice is that you can immediately see which parts of the system are broken by your change immediately as you…

I agree it's definitely an underserved tooling area. But so is a maze of submodules which is the realistic alternative. Open source just doesn't care too much about whole-company level projects because those don't really exist in the open source world. That said, you can definitely make it work with Git & Bazel. Beats submodules by a mile.

Oh yes.

Also another thing: often people end up with the worst of both worlds: one (or two!) big monorepos plus a swarm of many little repos. Now you only pay the price of the monorepo but rarely ever notice the benefits

Re: Debian Git Monorepo

#94

Post by riku_iki »

wongarsu wrote: post #78
benrutter wrote: post #64

Earlier quoted context omitted.

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

Doesn't google have a lot of internal tooling to make that monorepo actually work for them? The issue isn't so much that large monorepos don't work, it's that you are trading one set of issues for another set of issues. And most popular open source tooling (and I assume all debian-specific tooling) is built around solving the issues of many repositories rather than solving the issues of monorepos.

> Doesn't google have a lot of internal tooling to make that monorepo actually work for them?

probably tests coverage and workflow matters: you can work with monorepo if you can check how your change will impact rest of the ecosystem.

Re: Debian Git Monorepo

#95

Post by riku_iki »

tetromino_ wrote: post #73
samus wrote: post #65

Earlier quoted context omitted.

The stuff in Google's monorepo is mostly their own code, not snapshots from upstream.

Google's monorepo includes snapshots of everything upstream that Google's proprietary code depends on - starting from the kernel and glibc and core language toolchains, and all the way up to various open-source libraries and modules in c/python/java/whatever.

But 3p sources likely are for specific stable version? Linus can't break Google by submitting bug into linux kernel?

Re: Debian Git Monorepo

#96

Post by Archelaos »

mewpmewp2 wrote: post #59
Archelaos wrote: post #39

Earlier quoted context omitted.

I would not call this a "library", but a set of interrelated project with "shared moduls" instead. With a "library" I mean a piece of code where the publisher does know who is using it for what exact purpose. Of couse we could now fight over a proper general definition for "library", but this is how I would use the term in this context, because thee long term aspect is what really makes outsourcing such code meaningf…

The argument then is also that monorepo allows you to use this as "shared modules" instead of published "libraries". The point is you have some code that you want to reuse, and either within a monorepo or between multiple repositories.

The question is, when it is wise to do so. Changing the public interface of a "shared module" or published "library" comes with a liability. Code reuse is almost a no brainer in the case of a standard library that rarely or never changes. Using a permanently changing modul whose modifications provide no real benefits for your project makes no economic sense. The sweetspot is somewhere in between.

What the parent actually did when copying the library/module code was creating a sort of a a long-term support version for himself maintained by himself. His cost-benefit analysis told him that this is better than always trying to keep up with the changes of the library/modul.

My comment was aimed at setting the bar rather high for when to share reusable libraries. And even for my own modules I rather prefer copying code instead of reusing it in different contexts. There is a reason why the contexts differ and it is typically easier and more economic to adress this differences in its particular place instead of preparing for them in shared code.

The more up-stream a library/module is, the more general it must be, but the less liable to change it should be.

Re: Debian Git Monorepo

#98

Post by Tyr42 »

benrutter wrote: post #64
alkonaut wrote: post #58

I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB? To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third p…

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

I think chrome is got its own thing if I remember from my internship. But almost everything in one repo sounds right.

Re: Debian Git Monorepo

#100

Post by ants_everywhere »

wongarsu wrote: post #78
benrutter wrote: post #64

Earlier quoted context omitted.

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

Doesn't google have a lot of internal tooling to make that monorepo actually work for them? The issue isn't so much that large monorepos don't work, it's that you are trading one set of issues for another set of issues. And most popular open source tooling (and I assume all debian-specific tooling) is built around solving the issues of many repositories rather than solving the issues of monorepos.

yes, and a lot of it's open source.

The main issue IMHO is that git doesn't scale for a monorepo. A monorepo wants to spread a single virtual repo over a large number of devices, with no device having the entire repo. In contrast, git wants to do the opposite: it's built for a large number of devices to have the same copy of the entire repo.

But aside from tooling and needing a horizontally scalable VCS, there's also the cost and engineering resources required to keep the cluster running, available, and high performance.

Post reply on HN