Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepoize – Bash scripts for creating a monorepo out of smaller repos

github.com

Post reply on HN
Print view View original

41–50 of 103 posts

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#41

Post by _se »

cryptica wrote: post #11
stepbeek wrote: post #10

Earlier quoted context omitted.

> Having separate repos only makes sense if they are maintained by separate teams with a well defined API Agree! I've found that a multirepo setup only works if there's an obvious boundary. Similar to SOA, distributed monoliths - or distributed monorepos - are problematic.

> Having separate repos only makes sense if they are maintained by separate teams with a well defined API If some logic is modular then it should be in a separate repo. If it's not modular then it should be part of the same repo; just regular files inside directories... IMO, monorepos try to have it both ways but this doesn't make any sense; it encourages developers to write logic which is "somewhat modular". But "so…

This is a ridiculous oversimplification that doesn't scale at all to larger orgs. Every modular component in its own repo? That basically means creating a repo for every library. You could easily be managing thousands of repos with negative benefit.

People seem to be afraid of using a single repo for some reason. It's really simple though. You should have one repo per "release cycle" that you have. Anything meant to be deployed together should be in the same repo. Things not meant to be deployed together should be in different repos. With the right tooling, that is what solves most problems.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#42

Post by beagle3 »

hocuspocus wrote: post #31
beagle3 wrote: post #25

Earlier quoted context omitted.

Personally, informal survey of people I know who work at big corps. Google and Microsoft have both evaluated this internally and reached that conclusion, with some of the evaluation criteria and conclusions publicly documented (though I don’t have links available on my phone, google will likely find them for you)

Microsoft doesn't really use a monorepo. This leaves companies like Google and Facebook that made a decision fairly early in their existence, set constraints, and then spent several hundreds engineer-years into developing their own infrastructure and tools to support that decision. Does it work? Yes. Is it more efficient than what other companies of similar scale are doing? I don't believe anyone knows.

From reading articles, I got the impression Windows is one monorepo (but it doesn’t include Office) and e.g. Visual Stduio is another. So it’s true that Microsoft doesn’t use a monorepo as a whole, but e.g. just Windows itself is a codebase significantly larger and more diverse than 99% of companies would have, and it does have multiple “independents” products - Notepad is an independent module from Solitaire and they are both independent of essentially everything else, whereas e.g. File Explorer, the win32 subsystem and the kernel often need to be updated together for new features.

GoogBook made it early, but re-evaluated. For sure, they found the cost of switching is higher than any potential benefit. I must say that I have not found anyone describing benefits other than “git is faster for smaller repositories” (which was very true but is only slightly true with sparse checkouts and shallow branches these days) and “I like it that way”.

Yosefk has a convincing article about “if your culture is bad, it doesn’t matter; if your culture is good, monorepos are technically easier until you hit some scale wall but then they don’t become harder that multirepo at that scale” IIRc.

But w.r.t Microsoft I admit I’m not very well informed, thank you for setting the record straight.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#43

Post by sgn »

leethargo wrote: post #33
eithed wrote: post #30

Earlier quoted context omitted.

You can preserve history. You can also squash it.

Maybe I don't know about git subtree. I guess it can "preserve history" in the sense of keeping a corresponding new commit for every old commit, but they wouldn't have the same hash id?

They have everything the same with old subtree. From commit message, history, object ID, commit ID.

git-subtree(1) also allows splitting subtree.

git-subtree(1) uses plumbing git command, which is stable interface, unlike this one, which uses porcelain command.

IIRC, Git’s maintainer uses git-subtree(1) to merge gitk and git-gui to Git.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#44

Post by michaelmcmillan »

koonsolo wrote: post #21
frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

In git I fully agree, and I wonder which company successfully runs a monorepo in git. For me, I prefer git submodules, which seem to have the benefit of both monorepo and separate repo's.

> I wonder which company successfully runs a monorepo in git.

Google. They're not using git, but their own scm: https://research.google/pubs/pub45424/

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#45

Post by bluGill »

auxym wrote: post #26
frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture.

Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your answer is the right one for everyone else.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#46

Post by kdumont »

edwintorok wrote: post #6
stared wrote: post #5

In the last few years, I saw a lot of people using monorepos, and discouraging using submodules. Frankly, I don't know why. For small projects, sure - it it an overhead. For anything larger, I find it useful to encapsulate things - be it installable packages, or if it is not the case - at least other repositories. In the last months I split one project into 4 repos ( https://github.com/Quantum-Game/ ) and couldn't be…

Once your code grows and you end up with too many repos you'll be yearning to get back to monorepos. It allows to make changes in one go instead of having 3-4 PRs that all have to be merged at same time or otherwise the build breaks. Having separate repos only makes sense if they are maintained by separate teams with a well defined API in-between that doesn't change often. If the same team maintains all repos then it…

> It allows to make changes in one go instead of having 3-4 PRs that all have to be merged at same time or otherwise the build breaks.

Can't you solve this by pointing submodules in the parent repo to a specific commit SHA before the breaking change? Then update the pointer in parent repo when you've made the corresponding changes.

I understand it's still added work compared to a monorepo, but seems like this could be a controlled way to handle submodules when necessary.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#47

Post by bluGill »

_se wrote: post #41
cryptica wrote: post #11

Earlier quoted context omitted.

> Having separate repos only makes sense if they are maintained by separate teams with a well defined API If some logic is modular then it should be in a separate repo. If it's not modular then it should be part of the same repo; just regular files inside directories... IMO, monorepos try to have it both ways but this doesn't make any sense; it encourages developers to write logic which is "somewhat modular". But "so…

This is a ridiculous oversimplification that doesn't scale at all to larger orgs. Every modular component in its own repo? That basically means creating a repo for every library. You could easily be managing thousands of repos with negative benefit. People seem to be afraid of using a single repo for some reason. It's really simple though. You should have one repo per "release cycle" that you have. Anything meant to…

I have two things that are not deployed together. However there is common algorithms between the two that I'd rather not copy and paste maintain. Mono repo doesn't make sense because there is always something close to release and getting risk adverse (there are legal consequences to some bugs) so nothing can every be updated.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#48

Post by zaphar »

The single biggest impediment to monorepo's might be jenkins. Monorepos need a radically different way to do CI/CD and none of the existing tooling does it well. There is probably a market out there for some tooling to do good Monorepo CI/CD without requiring hacky scripts and polling.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#49

Post by neeeeees »

zaphar wrote: post #48

The single biggest impediment to monorepo's might be jenkins. Monorepos need a radically different way to do CI/CD and none of the existing tooling does it well. There is probably a market out there for some tooling to do good Monorepo CI/CD without requiring hacky scripts and polling.

This is really a build system issue - ie figuring out which parts of the (mono)repo are affected by which change. Bazel, for example, depends on explicit, per-directory, BUILD files, and does a fairly good job of finding dependants.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#50

Post by dan-robertson »

stared wrote: post #5

In the last few years, I saw a lot of people using monorepos, and discouraging using submodules. Frankly, I don't know why. For small projects, sure - it it an overhead. For anything larger, I find it useful to encapsulate things - be it installable packages, or if it is not the case - at least other repositories. In the last months I split one project into 4 repos ( https://github.com/Quantum-Game/ ) and couldn't be…

There’s a longer discussion here which may interest you: https://danluu.com/monorepo/

But if your code uses someone else’s libraries that don’t change or has super well defined interfaces between parts that never change or is going to be a mix of languages then maybe you won’t have issues or see and advantage of monorepos.

If you have a library in a monorepo then it is much easier to improve the library in a breaking way and upgrade all its dependencies. If you have many repos then this work gets more distributed (so one person does it many times or lots of people who are less expert in the change to the library have to do it) or libraries are just not frequently updated and you get the same helper functions not being up streamed or you use different interfaces for the same library in different repos.

Post reply on HN