Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepoize – Bash scripts for creating a monorepo out of smaller repos

github.com

Post reply on HN
Print view View original

51–60 of 103 posts

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#51

Post by cryptica »

_se wrote: post #41
cryptica wrote: post #11

Earlier quoted context omitted.

> Having separate repos only makes sense if they are maintained by separate teams with a well defined API If some logic is modular then it should be in a separate repo. If it's not modular then it should be part of the same repo; just regular files inside directories... IMO, monorepos try to have it both ways but this doesn't make any sense; it encourages developers to write logic which is "somewhat modular". But "so…

This is a ridiculous oversimplification that doesn't scale at all to larger orgs. Every modular component in its own repo? That basically means creating a repo for every library. You could easily be managing thousands of repos with negative benefit. People seem to be afraid of using a single repo for some reason. It's really simple though. You should have one repo per "release cycle" that you have. Anything meant to…

>> Every modular component in its own repo

My original statement was slightly misleading. I meant to say "every modular component which needs to be reused by different projects" obviously I'm not advocating that every class should be in a separate repo; that would be madness. My point is; if it's modular and needs to be reused in different projects, then it should be a separate repo, otherwise, it can just be added as files inside the current project repo; no need to publish those as separate standalone libraries for download. If the component is not modular, then it shouldn't be shared with any other projects.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#52

Post by pjmlp »

wikibob wrote: post #38
pjmlp wrote: post #36

Earlier quoted context omitted.

Microsoft does definitely use git, monorepo I am not so sure.

https://www.google.com/search?q=microsoft+monorepo https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

That doesn't say that everything related to Windows is developed on the same repo, just the kernel and several core components.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#53

Post by jedieaston »

koonsolo wrote: post #35
Kipters wrote: post #28

Earlier quoted context omitted.

> I wonder which company successfully runs a monorepo in git. Microsoft

For which product is this? It can't be all of them right?

Windows is in one repo at Microsoft, I believe.

https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#54

Post by wwright »

bluGill wrote: post #47
_se wrote: post #41

Earlier quoted context omitted.

This is a ridiculous oversimplification that doesn't scale at all to larger orgs. Every modular component in its own repo? That basically means creating a repo for every library. You could easily be managing thousands of repos with negative benefit. People seem to be afraid of using a single repo for some reason. It's really simple though. You should have one repo per "release cycle" that you have. Anything meant to…

I have two things that are not deployed together. However there is common algorithms between the two that I'd rather not copy and paste maintain. Mono repo doesn't make sense because there is always something close to release and getting risk adverse (there are legal consequences to some bugs) so nothing can every be updated.

The release cycle of that common code is also a big factor.

If your common code changes fairly irregularly, you are correct, monorepos have reduced benefit.

If your common code is as likely to change as anything else, or more likely, then your multirepo setup can begin to incur more cost.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#55

Post by dangoor »

koonsolo wrote: post #21
frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

In git I fully agree, and I wonder which company successfully runs a monorepo in git. For me, I prefer git submodules, which seem to have the benefit of both monorepo and separate repo's.

Khan Academy has a monorepo in git. With 10 years of history in it, it's not always pleasant but does come with some advantages.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#56

Post by auxym »

bluGill wrote: post #45
auxym wrote: post #26

Earlier quoted context omitted.

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your a…

> If that happens often you have the wrong repo split.

Yes, but: you pick out a nice, clean perfectly segregated architecture. Two years later, things have evolved, and a new feature that no one initially thought of is asked and involved refactoring 25 repos. Oops.

Side note: not my architecture. I only worked there for a short while. I have no skin in the game either way, I'm sure I agree with you that mono and multi- repos are the right solution for different problems.

Just trying to expose problems I've seen happen in a massive multirepo approach. Everyone thought it was the correct solution after being burned by a massive monolith that over two decades became an entangled mess. They thought they had the perfect architecture, the "correct repo split". In practice it's almost impossible, which is a major tradeoff to be considered.

Minimizin the number of repos and splitting only when absolutely necessary at least has a chance of reducing that complexity. The default should be "as few repos as possible", not "we're going multirepo, let us break this down into as many small parts as possible as our first step", which might be what the architects were used to in modular program architecture (small, reusable functions/classes).

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#57

Post by mumblemumble »

auxym wrote: post #26
frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

I worked in a company that had at least 100 repos, often with interdependencies, and was managing it well using svn.

Then we migrated to Git, which doesn't have a good equivalent to svn's externals, and things kind of went to heck for a while. We eventually hacked together an in-house system that brought back some of what we were missing, but it never worked as smoothly as svn externals did.

I was originally one of the advocates of moving to Git. Hindsight being 20/20, after the dust settled I realized we would have been fine staying on svn, anyway. With such a high number of individual repositories, blocking on each other and nasty merge conflicts were rarely a problem in practice because we had over 100 swim lanes clearly marked in the pool. Lots more swim lanes than we had developers.

FWIW, one team dealt with it by migrating just their stuff to their own monorepo, and I wasn't too impressed by that outcome, either. They started seeing an outsize rate of technical debt accumulation as things that previously were easier to keep well modularized started naturally getting more tangled together. I realize that is a problem that can easily be managed with just a little effort. But, like every other dev problem that can easily be managed with just a little effort, it wasn't.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#59

Post by ratww »

beagle3 wrote: post #16
cryptica wrote: post #13

Earlier quoted context omitted.

I think it's because corporations treat developers like untrustworthy and replaceable cogs in a big machine - IMO, this is what's wrong, not multi-repos. If corporations gave their developers responsibility and ownership over the maintenance of specific repos which are heavily depended upon, then multi-repos would not be a problem. It's a great source of pride for developers to feel responsible for something and to t…

I disagree about the nature of the problem. You can’t give sole ownership of a project people depend on to just one person - a bus factor of 1 is unacceptable in most circumstances. Busses happen, people leave, etc. Some things are also too big for one person. And when you give it to a team of proud capable developers, you will get ego wars, blame throwing, etc. with probability that increases quickly with team size.…

Giving ownership to a single person doesn't imply a bus factor of one. As long as someone can take over, it's not a problem.

I've had this happen lots of times in my career, and most of the time dealing with such projects was faster than onboarding in a company-owned repo by at least an order of magnitude. I really mean it: minutes instead of days to be productive.

In my experience most engineers have the best intentions and want to do the best work possible, until you force them into working in a bad environment where he's treated like... a cog in the machine with zero ability to influence his own productivity, happiness or even the quality of his code.

The biggest reason large companies "can't afford" those things is because they're hell bent on turning engineerings into replaceable cogs, out of ideology.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#60

Post by namelosw »

frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

It's about having boundaries that make sense. The best scenario is having multiple codebases, while when someone wants to change something, it can be done by just modifying code in one repo.

However, the worst scenario is also having multiple codebases, but when someone wants to change something, they end up having to go through many codebases and have to deal with all the environment setup, coordination, and possibly the worst of them all - the versions.

Monorepo is a straightforward way to avoid the best and worst scenario at the same time. So you'll give up the codebase splitting game and never worry about them, then focus on things that matter more.

Post reply on HN