Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepoize – Bash scripts for creating a monorepo out of smaller repos

github.com

Post reply on HN
Print view View original

31–40 of 103 posts

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#31

Post by hocuspocus »

beagle3 wrote: post #25
hocuspocus wrote: post #17

Earlier quoted context omitted.

> Because it turns out big corporations that use multirepos tend to be even less efficient (about versioning) than those that use monorepos. How do you measure that exactly?

Personally, informal survey of people I know who work at big corps. Google and Microsoft have both evaluated this internally and reached that conclusion, with some of the evaluation criteria and conclusions publicly documented (though I don’t have links available on my phone, google will likely find them for you)

Microsoft doesn't really use a monorepo.

This leaves companies like Google and Facebook that made a decision fairly early in their existence, set constraints, and then spent several hundreds engineer-years into developing their own infrastructure and tools to support that decision.

Does it work? Yes. Is it more efficient than what other companies of similar scale are doing? I don't believe anyone knows.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#32

Post by xorcist »

frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

On the contrary, you should provide a (good) argument before splitting a project. One such argument might be that those parts, for example configuration code, is on such a difference cadence that it should have a different branch and release model. Or that some parts must be kept unreadable for most developers. Or that the project has simply grown too large.

A good rule of thumb can be how much larger your project is than the Linux kernel itself. The kernel seems to constantly get near the number of objects that git can confortably handle without any major speedbumps.

Until then, do not bother. Be prepared that splitting a project in multiple repos will always require some sort of external tooling as soon as tickets or pull request workflows span multiple repos.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#33

Post by leethargo »

eithed wrote: post #30
leethargo wrote: post #29

Earlier quoted context omitted.

Git subtree wouldn't preserve hashes of the existing history?

You can preserve history. You can also squash it.

Maybe I don't know about git subtree. I guess it can "preserve history" in the sense of keeping a corresponding new commit for every old commit, but they wouldn't have the same hash id?

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#34

Post by dustinchilson »

Having done this migration, I recommend you look into filter-repo, https://github.com/newren/git-filter-repo

I don't remember the specifics but the method used here didn't produce the results we were looking for when migrating long histories and lots of branches.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#35

Post by koonsolo »

Kipters wrote: post #28
koonsolo wrote: post #21

Earlier quoted context omitted.

In git I fully agree, and I wonder which company successfully runs a monorepo in git. For me, I prefer git submodules, which seem to have the benefit of both monorepo and separate repo's.

> I wonder which company successfully runs a monorepo in git. Microsoft

For which product is this? It can't be all of them right?

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#36

Post by pjmlp »

Kipters wrote: post #28
koonsolo wrote: post #21

Earlier quoted context omitted.

In git I fully agree, and I wonder which company successfully runs a monorepo in git. For me, I prefer git submodules, which seem to have the benefit of both monorepo and separate repo's.

> I wonder which company successfully runs a monorepo in git. Microsoft

Microsoft does definitely use git, monorepo I am not so sure.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#37

Post by sokoloff »

ex_amazon_sde wrote: post #22
beagle3 wrote: post #9

Earlier quoted context omitted.

Because it turns out big corporations that use multirepos tend to be even less efficient (about versioning) than those that use monorepos. What are those far simpler, better, more open alternatives?

> it turns out big corporations that use multirepos tend to be even less efficient (about versioning) than those that use monorepos Amazon disagrees.

Does Amazon optimize for efficiency in software engineering, or just for the overall quantity of output? My sense, having spoken with Amazon SWEs and a fair number of engineering leaders is that they optimize for overall output, but not for efficiency (output per worker).

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#38

Post by wikibob »

pjmlp wrote: post #36
Kipters wrote: post #28

Earlier quoted context omitted.

> I wonder which company successfully runs a monorepo in git. Microsoft

Microsoft does definitely use git, monorepo I am not so sure.

https://www.google.com/search?q=microsoft+monorepo

https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#39

Post by beagle3 »

sokoloff wrote: post #37
ex_amazon_sde wrote: post #22

Earlier quoted context omitted.

> it turns out big corporations that use multirepos tend to be even less efficient (about versioning) than those that use monorepos Amazon disagrees.

Does Amazon optimize for efficiency in software engineering, or just for the overall quantity of output? My sense, having spoken with Amazon SWEs and a fair number of engineering leaders is that they optimize for overall output, but not for efficiency (output per worker).

Well, the measure of efficiency most of the world cares about is “overall profit”, which — around the efficient frontier — has a good proxy with “overall output”.

I feel your “output per worker” measure of average individual efficiency is not a good metric to optimize - just fire everyone except your best employee and you’ve maximized it.

Edit: employer -> employee typo

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#40

Post by sokoloff »

beagle3 wrote: post #39
sokoloff wrote: post #37

Earlier quoted context omitted.

Does Amazon optimize for efficiency in software engineering, or just for the overall quantity of output? My sense, having spoken with Amazon SWEs and a fair number of engineering leaders is that they optimize for overall output, but not for efficiency (output per worker).

Well, the measure of efficiency most of the world cares about is “overall profit”, which — around the efficient frontier — has a good proxy with “overall output”. I feel your “output per worker” measure of average individual efficiency is not a good metric to optimize - just fire everyone except your best employee and you’ve maximized it. Edit: employer -> employee typo

I think your first point is good and agree with it. Even within optimizing for that (rather than output/worker or output/salary), I think it's fair to examine whether specific choices are optimized for the same case that you are facing. Not everything done by very successful companies should be replicated in your business.
Post reply on HN