Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepo or Multirepo? Role-Based Repositories

blog.7mind.io

Post reply on HN
Print view View original

21–30 of 120 posts

Re: Monorepo or Multirepo? Role-Based Repositories

#21

Post by punkdata »

ChristianBundy wrote: post #15

Why not both? I've been using https://github.com/mateodelnorte/meta and having a great time so far, it's just that GitHub (and others) don't have a simple way to bundle multi-repo commits in pull requests.

I agree with Christian... Why not both? Lots of teams I interact with have great reasons for a monorepo which they admit requires some work in tooling and processes and claim they're successfully releasing software faster with less effort if their code lived in disparate repos. I believe teams must choose the appropriate patterns that work best for their architectures and situations.

Re: Monorepo or Multirepo? Role-Based Repositories

#22

Post by dmoy »

username90 wrote: post #14
Cedricgc wrote: post #3

The main driver of success in either model is in the tooling and practices invested in it to make it work in an organization. Google is successful with their monorepo because they have invested in building (blaze), source control (piper, code search), and commit to always developing on HEAD. Multirepo is currently easier for most companies because most public tooling (git, package manager) is built around multirepos.…

I wonder why nobody have made a good public monorepo offering similar to what Google have internally. Would probably be a hit at many companies since it fixes so many issues related to working in very large teams.

At large enough scale, it causes a lot of problems and breaks like every other dev tool requiring a lot of work to get it back together.

That said, there are some open source pieces to help. Facebook open sourced their mercurial stuffs so you can get version control at scale (and before then you just use perforce). Google open sourced bazel. Google open sourced some parts of the underlying infra behind code search, but not enough to really work properly. And of course lower level there's a plethora of reasonable db offerings, etc.

It would still require a lot of glue though.

Re: Monorepo or Multirepo? Role-Based Repositories

#23

Post by username90 »

weberc2 wrote: post #12
01100011 wrote: post #6

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do not prevent you from segmenting your code into modules and pushing binary/source packages into source control so that builds can avoid compiling everything(TiVo used to do this, and it worked well when you got the hang of it). I feel like these debates are ofte…

Can you elaborate on “monorepos do not prevent you from checking packages into source control” and how that helps to avoid recompiling everything? Why would you check a package into source control anyway? Surely source control is for source code? And I lean toward monorepos, btw, but there are still lots of obstacles and monorepo proponents don’t tend to acknowledge them or offer clear suggestions for how to solve or…

At Google we check in the source of every library into the monorepo and compile them ourselves with cached builds from a central server, I don't think we use package managers.

Re: Monorepo or Multirepo? Role-Based Repositories

#24

Post by jsnell »

01100011 wrote: post #6

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do not prevent you from segmenting your code into modules and pushing binary/source packages into source control so that builds can avoid compiling everything(TiVo used to do this, and it worked well when you got the hang of it). I feel like these debates are ofte…

VCS access control are the wrong tool for solving the "people use code they shouldn't" complaint.

First, VCS ACLs will massively reduce the benefits you're supposed to get from a monorepo. How will you do global refactors in that kind of a situation? How does a maintainer of a library figure out how the clients are actually using it? (The clients must have visibility into the library, but the opposite it unlikely to be true.)

Second, let's say that I maintain a library with a supported public interface that's implemented in terms of an internal interface that nobody's supposed to use. How will VCS ACLs allow me to hide the implementation but not the interface? When they kick off a build, the compiler needs to be able to read the implementation parts to actually build the library. It can't be that the clients have access to read the headers but then link against a pre-build binary blob. At that point you don't have a monorepo, you've got multirepos stored in a monorepo.

The actual solution are build system ACLs. Not ACLs for people, but ACLs for projects. Anyone can read the code, but you can say "only source files in directory X can include this header" or "only build files in directory Y can link against this object file".

Re: Monorepo or Multirepo? Role-Based Repositories

#25

Post by username90 »

dmoy wrote: post #22
username90 wrote: post #14

Earlier quoted context omitted.

I wonder why nobody have made a good public monorepo offering similar to what Google have internally. Would probably be a hit at many companies since it fixes so many issues related to working in very large teams.

At large enough scale, it causes a lot of problems and breaks like every other dev tool requiring a lot of work to get it back together. That said, there are some open source pieces to help. Facebook open sourced their mercurial stuffs so you can get version control at scale (and before then you just use perforce). Google open sourced bazel. Google open sourced some parts of the underlying infra behind code search, b…

It is just that Googles tooling around code works really well together. Code search to view code and directory based history so you aren't swamped by others commits, tap to run all unit tests all the time but with sectioned projects so you don't run all tests on every presubmit, sponge to gather every test log ever (even for the tests you run locally) so you can link full test logs to coworkers when you have problems (also note that the logs source file links are actual links into code search), critique for easy versioned code reviews where you can diff code between any sets of comments so you can see its evolution and running presubmit checks and sponge links for tests, blaze to make a structured directory dependency management system to make partial checkouts and distributed cached builds work well.

I'd like a set of tightly coupled tool like that working outside of Google, but I guess it might be a just a dream, it is a bit too big of a project.

Re: Monorepo or Multirepo? Role-Based Repositories

#26

Post by megous »

pricechild wrote: post #2

I regularly join projects where someone has decided to place the project's code in half a dozen different repositories. Even though it's one project. Even though they refuse to allow a release of a single component - it must all be released together without forwards/backwards compatibility. I think most of of the time, the mono/multi debate is spoiled by people who feel they can have their cake and eat it too.

I think that whether to use mono/multi repo depends on whether you're willing to dump money into updating everyting at once, or not. If not, monorepos are really a big hindrance. It's better to split on the project boundary (things that may have different development paces), and use git worktree for having different versions of libraries checked out for building/bundling.

It works fairly nicely with meson, as you can simply checkout a worktree of a library into a subprojects directory, and let individual projects move at their own paces even if you don't do releases for the libraries/common code.

It's not really clear why having to update every consumer in sync with library changes is beneficial. Some consumers might have been just experiments, or one off projects, that don't have that much ongoing value to constantly port them to new versions of the common code. But you may still want to get back to them in the future, and want to be able to build them at any time.

It's just easier to manage all this with individual repos.

Re: Monorepo or Multirepo? Role-Based Repositories

#27

Post by scarmig »

I'm curious: how would most people here define monorepo vs multirepo?

On the surface, most people seem to think of a monorepo as a source control management system that exposes all source code as if it's a traditional filesystem accessed through a single point of entry. Multirepo, in contrast, seems to be about multiple points of entry.

But that's a superficial and uninteresting distinction. All the hard parts of managing code remain for both and, for a sufficiently large organization, you'll still need multiple dedicated teams to build tooling to make either work at scale. All the pros listed in the article need a team to make them work for either approach, and all the cons are a sign that you need a team to be make up for that deficiency for either approach.

Aesthetically a single point of entry appeals to me, in that it allows for a more consistent interface to code. But I'd go for good tooling above that in a heartbeat.

Re: Monorepo or Multirepo? Role-Based Repositories

#28

Post by jayd16 »

username90 wrote: post #23
weberc2 wrote: post #12

Earlier quoted context omitted.

Can you elaborate on “monorepos do not prevent you from checking packages into source control” and how that helps to avoid recompiling everything? Why would you check a package into source control anyway? Surely source control is for source code? And I lean toward monorepos, btw, but there are still lots of obstacles and monorepo proponents don’t tend to acknowledge them or offer clear suggestions for how to solve or…

At Google we check in the source of every library into the monorepo and compile them ourselves with cached builds from a central server, I don't think we use package managers.

How do you track dependencies of dependencies. Do you need to manually add the full dependency tree and re implement the dependency tracking through your internal system? If a project uses maven or gradle, you need to rewrite those files to point to your internal builds instead?

Re: Monorepo or Multirepo? Role-Based Repositories

#29

Post by jayd16 »

Whats the current state of git submodules? It seems like you could get some of the benefits of mono-repos in that you can reference dependency projects directly like a mono-repo. You can, in theory, treat many projects like a single code base.

I don't see it used very often though. Why not?

Re: Monorepo or Multirepo? Role-Based Repositories

#30

Post by minxomat »

chrisseaton wrote: post #7
alexnewman wrote: post #5

Earlier quoted context omitted.

Google uses no versioned libraries?

I think they do, but there is only ever one single version in use - the version in the repo.

Yes, and that prevents duplication of work when it's time to upgrade: https://github.com/microsoft/TypeScript/issues/33272
Post reply on HN