Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the pros / cons of using monorepos?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Ask HN: What are the pros / cons of using monorepos?

#81

Post by ptsneves »

qznc wrote: post #68
KerrickStaley wrote: post #63

Earlier quoted context omitted.

Git + Bazel will be fine for scaling up to 100s of engineers from my experience (at Lyft's L5 autonomous division). My other data point is Google, with 10,000s of engineers and a bespoke VCS. I'm not sure how things work in the middle (1000s of engineers), but I think you can solve that problem when you get to it, and Bazel has some features (look at the git_repository rule) to help you split a big repo if you need t…

I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significan…

Have you considered making shallow clones? Also indeed you are doing many things that harm usability. I am surprised though you are saying you have issues with blobs even in LFS.

Re: Ask HN: What are the pros / cons of using monorepos?

#82

Post by KerrickStaley »

qznc wrote: post #68
KerrickStaley wrote: post #63

Earlier quoted context omitted.

Git + Bazel will be fine for scaling up to 100s of engineers from my experience (at Lyft's L5 autonomous division). My other data point is Google, with 10,000s of engineers and a bespoke VCS. I'm not sure how things work in the middle (1000s of engineers), but I think you can solve that problem when you get to it, and Bazel has some features (look at the git_repository rule) to help you split a big repo if you need t…

I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significan…

I would avoid Git submodules; it seems like a fairly poorly designed feature and I'm constantly getting my submodules into a bad state (e.g. when switching branches). The alternative that has worked a lot better for me, in repos that use Bazel, is Bazel external repositories (using git_repository or http_archive). A more direct replacement that I hear is better is git subtree although I haven't used it.

Also, you're probably already doing this, but make sure you're running Git on an SSD! Spinning rust will slow down VCS workloads, which are disk intensive, considerably.

Re: Ask HN: What are the pros / cons of using monorepos?

#83

Post by jiveturkey »

This isn't answerable in short form comment thread like you would find here.

Besides the structure of the repo itself, you have to consider the "base" SCM software, as well as the interfaces that are and can be built around it.

You are probably limiting yourself to a specific implementation of git, and for that, a specific and relatively concise answer can probably be given. But the issue in general is much, much more expansive than that. The question as posed doesn't give us enough information to give a proper answer. As such, you can see how the answers so far are all over the board.

Re: Ask HN: What are the pros / cons of using monorepos?

#84

Post by qznc »

gen220 wrote: post #74
qznc wrote: post #68

Earlier quoted context omitted.

I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significan…

Interesting... when you say fetch, are you talking about a cold-fetch? `du -sh repo` for me is 15GB, and we don't experience the slowness you describe. If I'm checking out a branch that diverged from master 1 year ago, it takes maybe... 30 seconds? But I don't know, maybe we've cast different git-spells from you all? We don't use sub-modules, for example.

I assume to some degree our Bitbucket server is slow. Not sure if Bitbucket is the culprit or our IT.

One the other hand, git-status is also slow like 30 seconds. That is something I rather attribute to Windows. Doing the same inside WSL is faster.

Some people invested quite some time to speed it up. So apparently it isn't some low hanging fruit.

Re: Ask HN: What are the pros / cons of using monorepos?

#85

Post by simonaco »

Pros:

1. single source code location 2. easier dependency mgmt 3. related changes can go in the same commit

Cons: 1. abstractions and boundaries may diverge across packages 2. build times can become unpredictable 3. CI can be unwieldy to set up

The creator of Nx Tools wrote about this recently: https://aka.ms/createmonorepo

Re: Ask HN: What are the pros / cons of using monorepos?

#86

Post by qznc »

ptsneves wrote: post #81
qznc wrote: post #68

Earlier quoted context omitted.

I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significan…

Have you considered making shallow clones? Also indeed you are doing many things that harm usability. I am surprised though you are saying you have issues with blobs even in LFS.

Yes.

Shallow clones and submodules do not go well together though: For shallow clones, git is setup to not track all remote branches. This can result in an error message when the submodule has not fetched the commit you want to switch to.

Re: Ask HN: What are the pros / cons of using monorepos?

#87

Post by qznc »

KerrickStaley wrote: post #82
qznc wrote: post #68

Earlier quoted context omitted.

I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute. To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significan…

I would avoid Git submodules; it seems like a fairly poorly designed feature and I'm constantly getting my submodules into a bad state (e.g. when switching branches). The alternative that has worked a lot better for me, in repos that use Bazel, is Bazel external repositories (using git_repository or http_archive). A more direct replacement that I hear is better is git subtree although I haven't used it. Also, you're…

Yes, I use an SSD.

Our next try is probably to rely more on Conan and Artifactory to download binary artifacts. In theory, that could result in similar behavior to Bazel cloud builds though not as integrated and granular.

Re: Ask HN: What are the pros / cons of using monorepos?

#88

Post by lallysingh »

nendroid wrote: post #62
lallysingh wrote: post #41

Earlier quoted context omitted.

Good point. I've gotten hit by this in monorepos but I probably just tried to merge instead of rebase. Both are crazy painful on large source bases. It still takes forever! I should replace that with 'most commits in history are irrelevant to you, so you have to dig smarter to understand what's been going on with your components'

Invest in using a gui merge tool. It makes this process 10x easier. Also your complaint is unrelated to monorepos vs. multiple repos. Additionally rebase or merge won't make the conflict issue go away. I recommend companies stop using rebase as it produces an in accurate history of what's going on in your git history.

I'm a magit-er now. In multiple repos, the history for one repo is limited to that repo, which is usually more topical for people working on it.

If I've got changes on an old version of the repo, and try to merge against a new one, I find conflicts in the changes others have made to the same repo -- I have the 'before' version, they have the 'after'. I frankly don't investigate too much when it hits.

As for history, I love rebase for two reasons: (1) I can squash mistakes in my history so they're gone forever. (2) In reconciliation I'm watching my changes filter through the history of the repo. When I understand what's up, this is great. When I don't, it's a catastrophe of me re-applying changes that haven't actually been applied yet, then conflicting with their application later. Ugh.

Re: Ask HN: What are the pros / cons of using monorepos?

#89

Post by qznc »

dastbe wrote: post #72
influx wrote: post #33

Twitter: Monorepo Facebook: Monorepo Google: Monorepo Amazon: Multirepo Of these companies, compare AWS and the number of services, new features and quick turn around time with their competitors, and get back to me.

What makes Amazon's multirepo setup "tick" isn't the multirepos themselves but the coordination mechanisms around it (version sets, brazil, and pipelines). whereas bazel/buck are open source and work ok enough to scale monorepos, i dont think there are the equivalent tools out there for multirepo and it shows in the comments for people with multirepo pain. EDIT: that being said, I do think multirepo is better, but on…

After some search I found this description, which seems to explain brazil and version sets: https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6...

Re: Ask HN: What are the pros / cons of using monorepos?

#90

Post by gregcoombe »

nimblegorilla wrote: post #71
gregcoombe wrote: post #8

Google posted a paper detailing their reasons for choosing monorepo: https://research.google/pubs/pub45424/ Caveat: your company probably isn't Google, so your challenges may be different.

Thanks for sharing that. It's mind-blowing that Google runs everything from a single repo. Most places that I work with create several repos for just one project.

Yeah, it was a significant infrastructure cost. I heard at one time that the single largest computer at Google was the Perforce server. They ended up completely re-writing it (called "Piper") for scaling. This is sort of what I was alluding to with my comment about Google having different concerns than many other companies. They can afford to dedicate a number of engineers to maintaining a monorepo system and then re-writing when it doesn't scale. That said, I personally believe that there are a lot of benefits to monorepos, and I think those tradeoffs are worth it for other companies too.
Post reply on HN