Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepoize – Bash scripts for creating a monorepo out of smaller repos

github.com

Post reply on HN
Print view View original

61–70 of 103 posts

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#61

Post by wst_ »

xorcist wrote: post #32
frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

On the contrary, you should provide a (good) argument before splitting a project. One such argument might be that those parts, for example configuration code, is on such a difference cadence that it should have a different branch and release model. Or that some parts must be kept unreadable for most developers. Or that the project has simply grown too large. A good rule of thumb can be how much larger your project is…

Depends what are you working on. Kernel is nothing alike enterprise environment. If you have services in your architecture, deployed separately, that is perfect argument to split your repository.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#62

Post by oconnor663 »

leethargo wrote: post #27
frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

Often there are implicit dependencies between (versions of) the many repos. E.g. where I work, it was decided to put test data in a separate from the code (to keep the latter repo small in size). But now, when you add a new test case, it will fail on older versions of the code. With a monorepo, you can always check out a consistent version of all parts.

And when your CI fails, it blames to a specific commit, rather than "the commit that triggered it, plus any commit in a dependency repo around the same time." And if you need to maintain old release branches, each one is a single branch, and your tooling doesn't need to know anything special about what branches of other repos to check out. And `git bisect` works.

All of these things are possible without a monorepo, by using higher-level tooling to manage the relationship between repos. It could be `git submodules`, or Repo, or whatever. But all of those things have their own downsides.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#63

Post by pierrebai »

mumblemumble wrote: post #57
auxym wrote: post #26

Earlier quoted context omitted.

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

I worked in a company that had at least 100 repos, often with interdependencies, and was managing it well using svn. Then we migrated to Git, which doesn't have a good equivalent to svn's externals, and things kind of went to heck for a while. We eventually hacked together an in-house system that brought back some of what we were missing, but it never worked as smoothly as svn externals did. I was originally one of t…

git has submodules.

A repo can thus host any number of sub-repos. Using modules can also be pretty much like having a monorepo if you have a master repo with all others under it. This way, you can still publish things like public API as a repo.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#64

Post by beagle3 »

bluGill wrote: post #45
auxym wrote: post #26

Earlier quoted context omitted.

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your a…

> you need to manage the problems of monorepos

If have not seen a problem on monorepos that does not exist on multirepos, other than more quickly reaching tools' repo size limits (but this is becoming less of a problem all the time on e.g. git and mercurial, and supposedly was never a problem with perforce).

I have seen many problems multirepos have that monorepos do not - such as that multirepos can (and almost always do) have a wrong split, and monorepos by definition cannot.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#65

Post by Kipters »

pjmlp wrote: post #36
Kipters wrote: post #28

Earlier quoted context omitted.

> I wonder which company successfully runs a monorepo in git. Microsoft

Microsoft does definitely use git, monorepo I am not so sure.

There have been many posts in their tech blogs about why they chose a monorepo

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#66

Post by vorpalhex »

auxym wrote: post #56
bluGill wrote: post #45

Earlier quoted context omitted.

If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your a…

> If that happens often you have the wrong repo split. Yes, but: you pick out a nice, clean perfectly segregated architecture. Two years later, things have evolved, and a new feature that no one initially thought of is asked and involved refactoring 25 repos. Oops. Side note: not my architecture. I only worked there for a short while. I have no skin in the game either way, I'm sure I agree with you that mono and mult…

> The default should be "as few repos as possible"

In any corporate/enterprise-y environment with multiple teams, the default should be the smallest piece likely to change ownership.

When we see a pattern where a feature involves refactoring a bunch of repos, then your initial system breakdown was horrendously off-track and you lost sight of your domains. Been there, it is fixable but it takes a bit.

> Everyone thought it was the correct solution

There is no correct solution, there is the best solution for your problem set. Sometimes that's a monolith, and sometimes that's several small repos. Every once in a while, it's even a monorepo.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#67

Post by pierrebai »

oconnor663 wrote: post #62
leethargo wrote: post #27

Earlier quoted context omitted.

Often there are implicit dependencies between (versions of) the many repos. E.g. where I work, it was decided to put test data in a separate from the code (to keep the latter repo small in size). But now, when you add a new test case, it will fail on older versions of the code. With a monorepo, you can always check out a consistent version of all parts.

And when your CI fails, it blames to a specific commit, rather than "the commit that triggered it, plus any commit in a dependency repo around the same time." And if you need to maintain old release branches, each one is a single branch, and your tooling doesn't need to know anything special about what branches of other repos to check out. And `git bisect` works. All of these things are possible without a monorepo, b…

git submodules tracks the version of the sub repos.

So if you organize that your releases are a master repo with all needed repos as sub-modules, your sub-repo version tracking is already done fr you.

All that while still allowing sub-repo to move forward faster than other repos. With a mono-repo, you can only allow some sub-system to go forward faster than others by making them live permanently in separate branches.

Basically, in monorepo you have to use branches to do what separate repo do normally for free.

(Then there is the issue that with a monorepo, any screw up screws everybody. You're all on the same boat, all the time.)

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#68

Post by beagle3 »

ratww wrote: post #59
beagle3 wrote: post #16

Earlier quoted context omitted.

I disagree about the nature of the problem. You can’t give sole ownership of a project people depend on to just one person - a bus factor of 1 is unacceptable in most circumstances. Busses happen, people leave, etc. Some things are also too big for one person. And when you give it to a team of proud capable developers, you will get ego wars, blame throwing, etc. with probability that increases quickly with team size.…

Giving ownership to a single person doesn't imply a bus factor of one. As long as someone can take over, it's not a problem. I've had this happen lots of times in my career, and most of the time dealing with such projects was faster than onboarding in a company-owned repo by at least an order of magnitude. I really mean it: minutes instead of days to be productive. In my experience most engineers have the best intent…

> In my experience most engineers have the best intentions and want to do the best work possible

I agree, most of them do - but I've also been in a place that highers the best of the best, and despite the best intentions, some people cannot put their ego aside and the overall result is between less-than-ideal and downright-horrible

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#69

Post by vorpalhex »

jedieaston wrote: post #53
koonsolo wrote: post #35

Earlier quoted context omitted.

For which product is this? It can't be all of them right?

Windows is in one repo at Microsoft, I believe. https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

Yeah but they basically wrote their own wrapper around it, which is to say Microsoft has fallen into one of their usual patterns:

1. Picking a trendy tool

2. Mis-using trendy tool

3. Rewriting trendy tool so it's no longer trendy tool but something custom and not quite standards-compliant with it's own weird behaviors and bugs

4. Complain the standard is wrong

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#70

Post by ratww »

beagle3 wrote: post #68
ratww wrote: post #59

Earlier quoted context omitted.

Giving ownership to a single person doesn't imply a bus factor of one. As long as someone can take over, it's not a problem. I've had this happen lots of times in my career, and most of the time dealing with such projects was faster than onboarding in a company-owned repo by at least an order of magnitude. I really mean it: minutes instead of days to be productive. In my experience most engineers have the best intent…

> In my experience most engineers have the best intentions and want to do the best work possible I agree, most of them do - but I've also been in a place that highers the best of the best, and despite the best intentions, some people cannot put their ego aside and the overall result is between less-than-ideal and downright-horrible

> ...until you force them into working in a bad environment where he's treated like... a cog in the machine with zero ability to influence his own productivity, happiness or even the quality of his code.
Post reply on HN