Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the pros / cons of using monorepos?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 100 posts

Re: Ask HN: What are the pros / cons of using monorepos?

#61

Post by gen220 »

qznc wrote: post #57
gen220 wrote: post #46

I've worked in environments across the version-control gamut. The best-run places have had monorepos. But, for the life of me, I would not trust the companies that didn't have monorepos, to operate a monorepo. To go mono is to make an org-level engineering/cultural commitment, that you're going to invest in build tools, dependency graph management, third-party vendoring, trunk-driven development, and ci/cd infrastruc…

What would be a scalable open-source monorepo stack? Git, Gitlab, Bazel?

This is more or less what we use at my current place (100s of engineers). We haven't run into any dramatic issues at this scale.

Our architecture is arcanist/phab over git --> stash (git) --> custom build server that's basically identical to gitlab --> artifactory.

we use pants, but we made that choice before bazel was a thing.

Bazel is probably the most "scalable" solution, but it isn't always the most intuitive/developer-friendly. It's worth exploring the alternatives (biggest ones imo are pants, buck) to see what you like the most. They're all pretty similar, but Bazel has some hard-to-replicate bells and whistles that mean it'll probably be the eventual winner.

Re: Ask HN: What are the pros / cons of using monorepos?

#62

Post by nendroid »

lallysingh wrote: post #41
ht85 wrote: post #26

Earlier quoted context omitted.

> Churn from other dev's stuff gets in your merge/rebase work Wouldn't other people' work only cause issues if they are changing the same files, in which case conflicts would happen even if the work is spread in multiple repos?

Good point. I've gotten hit by this in monorepos but I probably just tried to merge instead of rebase. Both are crazy painful on large source bases. It still takes forever! I should replace that with 'most commits in history are irrelevant to you, so you have to dig smarter to understand what's been going on with your components'

Invest in using a gui merge tool. It makes this process 10x easier. Also your complaint is unrelated to monorepos vs. multiple repos.

Additionally rebase or merge won't make the conflict issue go away. I recommend companies stop using rebase as it produces an in accurate history of what's going on in your git history.

Re: Ask HN: What are the pros / cons of using monorepos?

#63

Post by KerrickStaley »

qznc wrote: post #57
gen220 wrote: post #46

I've worked in environments across the version-control gamut. The best-run places have had monorepos. But, for the life of me, I would not trust the companies that didn't have monorepos, to operate a monorepo. To go mono is to make an org-level engineering/cultural commitment, that you're going to invest in build tools, dependency graph management, third-party vendoring, trunk-driven development, and ci/cd infrastruc…

What would be a scalable open-source monorepo stack? Git, Gitlab, Bazel?

Git + Bazel will be fine for scaling up to 100s of engineers from my experience (at Lyft's L5 autonomous division). My other data point is Google, with 10,000s of engineers and a bespoke VCS. I'm not sure how things work in the middle (1000s of engineers), but I think you can solve that problem when you get to it, and Bazel has some features (look at the git_repository rule) to help you split a big repo if you need to.

You may also want a service like Artifactory for hosting binary blobs that feed into the build process.

Re: Ask HN: What are the pros / cons of using monorepos?

#64

Post by taeric »

jrockway wrote: post #54
dehrmann wrote: post #40

Earlier quoted context omitted.

In mono/mulirepo debates, I frame this as multirepos make hard, scary things hard. Monorepos make it easy to introduce changes that break APIs at deploy time, but are hidden by the atomic commit actually being OK.

I think API versioning is orthogonal to how you organize your code. If you have multiple repos, then you still have that "atomic commit" where you update the version of the client library and adjust the code to handle the new semantics. It then breaks when you deploy it to production because the server code isn't deployed yet (or vice versa). Basically, if you make RPCs, breaking up your repositories or combining you…

I think they were referring to a false atomic commit to the front and back end. In a mono repo, that can be one commit. However, it has to be two deploys, so it should have been two commits.

At least, that is my argument. So, I could be reading my view into it. :)

Re: Ask HN: What are the pros / cons of using monorepos?

#65

Post by user5994461 »

dehrmann wrote: post #52

Monorepos: - Work best when you have an open culture - PCI compliance will be annoying - The obvious--everything in one place - Need good tooling around keeping master building - As they grow, become an uphill battle to use with an IDE - Test are likely to slow down as the repo grows, so tooling around tests - Usually lead to a rats nest of dependencies - Third-party library upgrades can be painful - Coupled with CD…

>> - PCI compliance will be annoying

I have to laugh at that. The two biggest banks Goldman Sachs and JP Morgan are heavily mono repo.

It actually makes all certifications and auditing easier. The shared tooling/platform can be checked, everything else can ride on it. Half the questions of certifications are about tracking changes... easy when it's all tracked by the repo.

Re: Ask HN: What are the pros / cons of using monorepos?

#66

Post by nendroid »

quicklime wrote: post #60

I won't try to be exhaustive here, but I think it's worth mentioning a few things: One con is that most open source (as well as publicly available but proprietary) tooling is geared towards the non-monorepo approach. So if you want to use a monorepo, you're going to have to fight a bit of an uphill battle because most tools and processes assume you have lots of little repos. For example, build triggers in a lot of CI…

Regression tests across multiple repos are still required. I don't see how splitting a monorepo into multiple repos gets rid of those tests.

Re: Ask HN: What are the pros / cons of using monorepos?

#67

Post by trixie_ »

Communitivity wrote: post #15

Mono-repos are either a shortcut to avoid release management and dependency management, or are a way to manage development at scales approximating Google's. First, for almost all companies copying non-selectively what Google does is harmful - you are not Google. Second, if you are a small team, your code only produces one binary that is shared across team boundaries, then you might be able to do with a monorepo. But…

Avoiding release and dependency management is not a shortcut - it's avoiding a needless detour. Management is overhead, overhead requires resources like the 'good devops' people you mention. Many companies and teams are lean and must find ways to work more efficiently - mono repos are simpler so they are the default option. Only if you have a really good reason to support multiple versions of software simultaneously or have real independent teams with boundaries should you consider making your process more complicated by breaking this up.

"The first rule of distributed systems is don’t distribute your system until you have an observable reason to."

Re: Ask HN: What are the pros / cons of using monorepos?

#68

Post by qznc »

KerrickStaley wrote: post #63
qznc wrote: post #57

Earlier quoted context omitted.

What would be a scalable open-source monorepo stack? Git, Gitlab, Bazel?

Git + Bazel will be fine for scaling up to 100s of engineers from my experience (at Lyft's L5 autonomous division). My other data point is Google, with 10,000s of engineers and a bespoke VCS. I'm not sure how things work in the middle (1000s of engineers), but I think you can solve that problem when you get to it, and Bazel has some features (look at the git_repository rule) to help you split a big repo if you need t…

I work in a project with 50 engineers. Our git repo is 2GB plus 5GB of submodules. Git is painfully slow and I would imagine with more engineers it would become unusable. For example, git-fetch takes at least a minute.

To some degree we are doing it wrong like using Windows, creating too many tags, and committing binary blobs (despite LFS). Still, scaling it up by factor of two or three would not change it significantly.

Re: Ask HN: What are the pros / cons of using monorepos?

#69

Post by user5994461 »

GuB-42 wrote: post #55
ht85 wrote: post #26

Earlier quoted context omitted.

> Churn from other dev's stuff gets in your merge/rebase work Wouldn't other people' work only cause issues if they are changing the same files, in which case conflicts would happen even if the work is spread in multiple repos?

Here is the problem: 1- You finished your work, so you pull from the central repository, and merge your your branch to the master 2- You do your merge work, test it a bit and commit 3- Now, you push and oops, another team did the same, you are now left with to heads 4- You merge or rebase the two heads. Probably a simple task, but you may still need to run some tests again. And if you are lucky, you are done, otherwi…

The CI should run tests and take care of merging to master, only after the tests passed. This avoids most problems.

Re: Ask HN: What are the pros / cons of using monorepos?

#70

Post by seg_lol »

jrockway wrote: post #54
dehrmann wrote: post #40

Earlier quoted context omitted.

In mono/mulirepo debates, I frame this as multirepos make hard, scary things hard. Monorepos make it easy to introduce changes that break APIs at deploy time, but are hidden by the atomic commit actually being OK.

I think API versioning is orthogonal to how you organize your code. If you have multiple repos, then you still have that "atomic commit" where you update the version of the client library and adjust the code to handle the new semantics. It then breaks when you deploy it to production because the server code isn't deployed yet (or vice versa). Basically, if you make RPCs, breaking up your repositories or combining you…

> API versioning is orthogonal to how you organize your code

They are, but they aren't, because we have to basically increment two counters when things change. If you want to be statically correct, you now have a distributed transaction to update both systems.

There is some path through a call graph (in-proc and rpc) that is statically valid. Then there are duck-typed results that are only additive, which you are describing. Provided the system can handle this (dictionaries over structs). You see this in protobuf and other static centric serialization systems. They start to encode dynamic information using the static building blocks. GCCs internal IR is like this. I think what you are advocating for only applies to the domain of microservices, loosely connected using languages and formats that are open under union. Other systems without those properties will not fare so well.

The folks using the monorepos probably don't disagree, but having atomic commits with hashes makes the identity function a useful crutch. I believe there are other solutions where the VCS could actually make this problem a non-issue.

Post reply on HN