Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debian Git Monorepo

blog.liw.fi

Post reply on HN
Print view View original

71–80 of 157 posts

Re: Debian Git Monorepo

#71

Post by xign »

BlackFly wrote: post #31
Cu3PO42 wrote: post #16

Earlier quoted context omitted.

There are real, tangible benefits to monorepos. And real, tangible downsides, too. Like everything else, it's a tradeoff. I'm sure people advocating for them have brought up some benefits. But in case they haven't, here's an example I recently encountered at work: for an application we are developing, we needed functionality X. We knew that we'd need X in other projects, too, so we made it a library. I published it t…

The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on tha…

I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. It really depends on how that's set up.

And how is sparse checkout worse for discoverability? With multiple repos it's even harder to find what you want sometimes if you are talking about 100's of random repos that aren't organized well.

Re: Debian Git Monorepo

#72

Post by bluGill »

Arch-TK wrote: post #22
Cu3PO42 wrote: post #16

Earlier quoted context omitted.

There are real, tangible benefits to monorepos. And real, tangible downsides, too. Like everything else, it's a tradeoff. I'm sure people advocating for them have brought up some benefits. But in case they haven't, here's an example I recently encountered at work: for an application we are developing, we needed functionality X. We knew that we'd need X in other projects, too, so we made it a library. I published it t…

I don't think this is really advocacy for monorepo but rather advocacy against premature solidification of abstractions. Whenever I encounter a situation like this, the first step I take is to write the code without any API or interface boundaries to worry about. Then if I still feel it would be a good idea to separate it, I will separate it while keeping it in the source tree (this isn't really vendoring or monorepo…

Most people arguing one way or the other are in such a small code base that they don't face the problems the argument is really about in the first place. If you only have a a million lines of code then by all means use a monorepo as you probably aren't facing the downsides that make people reach for a multi-repo solution anyway.

Re: Debian Git Monorepo

#73

Post by tetromino_ »

samus wrote: post #65
benrutter wrote: post #64

Earlier quoted context omitted.

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

The stuff in Google's monorepo is mostly their own code, not snapshots from upstream.

Google's monorepo includes snapshots of everything upstream that Google's proprietary code depends on - starting from the kernel and glibc and core language toolchains, and all the way up to various open-source libraries and modules in c/python/java/whatever.

Re: Debian Git Monorepo

#74

Post by sethammons »

gtirloni wrote: post #47
G3rn0ti wrote: post #7

I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?

> It’s a nightmare to handle and use Quite the opposite, IMHO. Juggling dozens of PRs across many repos for a simple change is the definition of a nightmare.

that tells me you prematurely split something. I worked at a high scale, high available system with multiple dozens of teams, each with several repos. If we had to sync changes across repos, we joined the repos because they were, well, joined. In the early days, we had shared libraries and keeping those in sync was a pita. Then we realized we should not be doing what we were doing.

The flow should be: (1) update a library and release it, preferring backwards compatible changes or a new major version; (2) code that wants/needs the new functionality updates the usage at its leisure. If you are having to keep multiple repos "in sync" with each other, yes, you picked the wrong abstraction. If the downstream repo(s) need to always be on the latest and greatest version of the library, the library code should be vendored with the calling code.

Re: Debian Git Monorepo

#75

Post by bluGill »

xign wrote: post #71
BlackFly wrote: post #31

Earlier quoted context omitted.

The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on tha…

I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing. It really depends on how that's set up. And how is sparse checkout worse for discoverability? With multiple repos it's even harder to find what you want sometimes if you are talking about 100's of random repos that aren't organ…

> I think in a centralized environment (workplace), it could be argued that immediately triggering all the build failures and having good hygiene in cleaning them up is actually not a bad thing.

In abstract I agree. However when I'm trying to get my code working having test failures in code that isn't even related to the problem I'm working on is annoying and I can't switch tasks to work on this new failure when the current code isn't working either.

Re: Debian Git Monorepo

#76

Post by sethammons »

surajrmal wrote: post #37
G3rn0ti wrote: post #7

I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?

I struggle to understand the upside of avoiding monorepos. A lot of people think it means you can't decouple releases, modularize builds and similar with a monorepo, but that's simply not true. The efficiency at which you can refactor and upgrade your codebase cannot be understated.

every monorepo I've seen suffers from a shit CI solution. Other teams break your builds. Builds get longer and more complex. Feedback loops slow down. Engineering velocity slows down. It works at Google because they spent, literally, hundreds of millions of dollars getting it working.

Re: Debian Git Monorepo

#77

Post by SOLAR_FIELDS »

jillesvangurp wrote: post #67
G3rn0ti wrote: post #7

I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?

Having lots of small repos also generates overhead they need to be checked out separately, tagged individually, built individually, etc. The reason that this is a bad April first joke is that the idea actually has merit. Google runs their company like this. The unthinkable bit is not doing this technically but Debian rearchitecting the way it works. Doing this with Git is a separate topic. It's probably not great for…

There was an article by some well known CTO that hits the nail on the head for me. Essentially they said that monorepo makes sense at small scale and enormous scale (eg Google) but anywhere in between that microservices make more sense. Knowing when you are going to be at those scales is key.

In any event, as has been alluded to in this thread already, any complaint about microservices vs monorepo and switching between them is almost always a tooling complaint. Companies will invest in one of these “philosophies” around code organization then won’t take the time to build the appropriate tooling to support them and then three years down the road some enterprising architect will switch from one to the other and the cycle begins anew

Re: Debian Git Monorepo

#78

Post by wongarsu »

benrutter wrote: post #64
alkonaut wrote: post #58

I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB? To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third p…

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt. Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

Doesn't google have a lot of internal tooling to make that monorepo actually work for them?

The issue isn't so much that large monorepos don't work, it's that you are trading one set of issues for another set of issues. And most popular open source tooling (and I assume all debian-specific tooling) is built around solving the issues of many repositories rather than solving the issues of monorepos.

Re: Debian Git Monorepo

#79

Post by Shacklz »

G3rn0ti wrote: post #7

I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?

There have already been a lot of points by other commenters, one point that I'd like to add:

It allows to solve "problems of scale" with technical solutions rather than process solutions.

For example: Let's say there's application app-A and app-B, and they decide they want a library lib-1 that they can share.

If they are in separate repositories, this means multiple pull-requests, it means separate pipelines where the pipeline of lib-1 likely won't include the tests of the applications, it means there will be pull-requests to the library which won't immediately be integrated into the applications so that some poor sod has to take care of breaking changes down the road, etc.

If they are in a monorepo, each application can set up tests that need to be fine with the library-code "as is", so any change to the library needs to work against existing application-tests. The price one pays however are pipelines that perform well - nobody wants to wait hours to get pipeline-feedback and such. A monorepo also allows ambitious folks to shine - it's easy to touch many things at once, or to touch things used by everyone, on a pull-request with high visibility, which is an easy platform to get "street creds" as an ambitious engineer.

Re: Debian Git Monorepo

#80

Post by ghosty141 »

BlackFly wrote: post #31
Cu3PO42 wrote: post #16

Earlier quoted context omitted.

There are real, tangible benefits to monorepos. And real, tangible downsides, too. Like everything else, it's a tradeoff. I'm sure people advocating for them have brought up some benefits. But in case they haven't, here's an example I recently encountered at work: for an application we are developing, we needed functionality X. We knew that we'd need X in other projects, too, so we made it a library. I published it t…

The real problems with monorepos are that most of the benefits vanish as the scale increases unless you invest into building more monorepo tooling. In your particular case, if your library becomes too popular, your one or two line implementation detail changes ripple out and trigger rebuilds of too many downstreams, many of which will have flakey tests and fail your MR. If most users are not actually depending on tha…

> Similarly, once you are too popular a library in a monorepo, you will never do any atomic breaking API changes since it would require updating too many downstreams.

This happens no matter which repo type. Even worse if a project chooses to update after a while, its far more painful having to do the changes after losing the context you had when you did the original changes.

If you want a monorepo, libraries being on the same version is a feature, and it keeps you from diverging.

Post reply on HN