Monorepo or Multirepo? Role-Based Repositories • Hacker News Board

Re: Monorepo or Multirepo? Role-Based Repositories

#51

Post by jumpingmice » Sat, Oct 12, 2019, 8:24 PM UTC

01100011 wrote: post #6

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do not prevent you from segmenting your code into modules and pushing binary/source packages into source control so that builds can avoid compiling everything(TiVo used to do this, and it worked well when you got the hang of it). I feel like these debates are ofte…

All of the supposed flaws of a monorepo in this article are actually flaws of git. This is a very common phenomenon. I often joke there are two kinds of developers: those who prefer monorepos and those who have never used perforce.

Re: Monorepo or Multirepo? Role-Based Repositories

#52

Post by likeliv » Sat, Oct 12, 2019, 8:25 PM UTC

avisser wrote: post #32

jayd16 wrote: post #29

Whats the current state of git submodules? It seems like you could get some of the benefits of mono-repos in that you can reference dependency projects directly like a mono-repo. You can, in theory, treat many projects like a single code base. I don't see it used very often though. Why not?

Even with submodules it's still a PR per repo. Global, atomic changes are super powerful.

There is another tool called git-subtree that should solve these problems, I think. But I've never seen it in use

Re: Monorepo or Multirepo? Role-Based Repositories

#53

Post by beagle3 » Sat, Oct 12, 2019, 8:26 PM UTC

swsieber wrote: post #16

jayd16 wrote: post #13

I don't understand the isolation difference. You can hide, protect and branch code in a monorepo so why is isolation a concern?

It depends on which VCS you use. Git for example, doesn't have any native support for hiding or protecting code in particular folders within the repository.

git has hooks for access control (which is how e.g. gitolite manages permissions, albeit at the whole repo level - I'm not familiar with an open-source hook that does directory level).

With respect to hiding, git has sparse checkouts that can give you a limited view of a repository (for performance reasons - not for security reasons)

But that's just today's git. Other VCSs like perforce provide much finer grained access control and hiding.

Re: Monorepo or Multirepo? Role-Based Repositories

#54

Post by weberc2 » Sat, Oct 12, 2019, 8:26 PM UTC

jayd16 wrote: post #28

username90 wrote: post #23

Earlier quoted context omitted.

At Google we check in the source of every library into the monorepo and compile them ourselves with cached builds from a central server, I don't think we use package managers.

How do you track dependencies of dependencies. Do you need to manually add the full dependency tree and re implement the dependency tracking through your internal system? If a project uses maven or gradle, you need to rewrite those files to point to your internal builds instead?

They use a tool called Blaze (Google around for “Bazel” which is the open source tool inspired by it). Basically you model the dependency tree such that the tool knows which targets are affected by a certain change, and then Blaze builds them in a clean room environment such that an undeclared dependency would cause the build to fail (hermetic builds). As far as I’m aware, this is the only way to sustainable operate a monorepo, but I would be happy to learn more if someone has other solutions.

Re: Monorepo or Multirepo? Role-Based Repositories

#55

Post by karmakaze » Sat, Oct 12, 2019, 8:37 PM UTC

pshirshov wrote: post #48

nhumrich wrote: post #46

The cons to multi repo are all anti patterns for microservices anyway. If you're doing microservices you shouldn't have build dependencies on other projects. The should only call eachother at a network level.

But... we do better things than microservices: https://github.com/7mind/slides/blob/master/02-roles/target/...

Read this, not worthwhile and OT.

Re: Monorepo or Multirepo? Role-Based Repositories

#56

Post by vogre » Sat, Oct 12, 2019, 8:40 PM UTC

jayd16 wrote: post #29

Whats the current state of git submodules? It seems like you could get some of the benefits of mono-repos in that you can reference dependency projects directly like a mono-repo. You can, in theory, treat many projects like a single code base. I don't see it used very often though. Why not?

Because if some change requires you to make commits to 3 repos, submodule approach will require to make 4 commits instead. Very annoying.

Re: Monorepo or Multirepo? Role-Based Repositories

#57

Post by jyounker » Sat, Oct 12, 2019, 8:59 PM UTC

pshirshov wrote: post #41

01100011 wrote: post #6

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do not prevent you from segmenting your code into modules and pushing binary/source packages into source control so that builds can avoid compiling everything(TiVo used to do this, and it worked well when you got the hang of it). I feel like these debates are ofte…

> Monorepo shortcomings 1 and 2 seem like bullshit to me. It's a blogpost and the author didn't try to build a total and exhaustive formal system. These shortcomings are not absolute truth but actually they are true. I've seen this multiple times: a small projects evolves over years into a monster. Engineers add new components and reuse any other components they may need creating horizontal links. At some point they…

Monorepo/multirepo and monolith/microservice are orthogonal concepts. When organizations don't understand that then they may end up building a distributed monolith in across multiple repos. (The "Distributed Big Ball of Mud" anti-pattern.)

Monorepo advocates are typically advocating for microservices, but within a single code base.

The way you provide access control is through code review and build system visibility.

In order to modify another group's code you require their approval on the review for that section of the code base. (Using mechanisms like github/gitlab owners files or rules within upsource.)

This still means that if one group needs to make extensive changes to another groups code, the path of least resistance may be to fork it into your own group's section of the repo.

Build tools provide another point of control. If you're using a tool like bazel, the way you link to a component in another portion of the repo is through target names. The only targets your code will have access to are those that the owners has declared as being available for external builds.

Re: Monorepo or Multirepo? Role-Based Repositories

#58

Post by jyounker » Sat, Oct 12, 2019, 9:07 PM UTC

jayd16 wrote: post #10

01100011 wrote: post #6

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do not prevent you from segmenting your code into modules and pushing binary/source packages into source control so that builds can avoid compiling everything(TiVo used to do this, and it worked well when you got the hang of it). I feel like these debates are ofte…

This is all true BUT I think the monorepo as described here is the act of treating all your projects as directly referencing each other. Sure you could just use a manyrepo style of dependency tracking in a monorepo but I think that's not exactly what the author is exploring.

> This is all true BUT I think the monorepo as described here is the act of treating all your projects as directly referencing each other.

From what I read that is a correct assessment. What the OP is proposing is something of a strawman argument. No advocate of monorepos I've ever met believe that a monorepo should imply a monolith.

Generally they're advocating monorepos in order to develop microservices faster, and with less effort. Using a monorepo and the associated tooling side steps the pain that comes from complicated CI, the difficulty of sharing code, the difficulties of non-atomic cross-repo reviews, and the difficulties of making multi-app refactorings.

Re: Monorepo or Multirepo? Role-Based Repositories

#59

Post by CuriousSkeptic » Sat, Oct 12, 2019, 9:10 PM UTC

weberc2 wrote: post #49

jayd16 wrote: post #43

Earlier quoted context omitted.

In short, the binaries are already built. Usually its faster to link to a prebuilt binary than to build from scratch.

So where do these binaries get built and how does the system know which binaries to rebuild for a given change? If developers are building binaries and committing them directly, doesn’t that open up security or even correctness issues? How does this approach satisfy compliance concerns (how can the CTO or a manager sign off on the changes that went into the binary if it’s just something a random developer committed?)…

Not sure how people do this in practice. But in principle it seems rather straight forward.

A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output.

If the box changes, run it in a trusted machine and save the output together with a signed declaration of which box version produced it

Re: Monorepo or Multirepo? Role-Based Repositories

#60

Post by Cedricgc » Sat, Oct 12, 2019, 9:12 PM UTC

username90 wrote: post #14

Cedricgc wrote: post #3

The main driver of success in either model is in the tooling and practices invested in it to make it work in an organization. Google is successful with their monorepo because they have invested in building (blaze), source control (piper, code search), and commit to always developing on HEAD. Multirepo is currently easier for most companies because most public tooling (git, package manager) is built around multirepos.…

I wonder why nobody have made a good public monorepo offering similar to what Google have internally. Would probably be a hit at many companies since it fixes so many issues related to working in very large teams.

Now that Git Virtual File System (VFS) is coming to GitHub, I think we shall see an uptick in monorepo adoption. Though what I said early about compatible tooling still applies. The repo style affects so much of the tooling, from Continous Integration, deployment, building, versioning, etc.

It really is not a small difference.