Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

31–40 of 268 posts

Re: The Ingredients of a Productive Monorepo

#31

Post by TeMPOraL »

makeitdouble wrote: post #26
AlotOfReading wrote: post #14

One thing I don't usually see discussed in monorepo vs multi repo discussions is there's an inverse Conway's law that happens: choosing one or the other will affect the structure of your organization and the way it solves problems. Monorepos tend to invite individual heroics among common infrastructure teams, for example. Because there are so many changes going in at once, anything touching a common area has a huge n…

Is your underlying assumption that the organization doesn't want to go one way or the other in the first place and is nudged by the technical choice afterwards ? I think most of the time the philosophical decision (more shared parts or better separation) is made before deciding how you'll deal with the repos. Now, if an org changes direction mid-way, the handling of the code can still be adapted without fundamentally…

I think GP's claiming it's a feedback loop, not one-directional relationship. Communication structure of an organization ends up reflected in the structure of systems it designs, and at the same time, the structure of a system influences the communication structure of the organization building it.

This makes sense if you consider that:

1) Changes to system structure, especially changes to fundamentals when the system is already being built, are difficult, expensive and time consuming. This gives system designs inertia that grows over time.

2) Growing the teams working on a system means creating new organizational units; the more inertia system has, the more sense it makes for growth to happen along the lines suggested by system architecture, rather than forcing the system to change to accommodate some team organization ideals.

Monorepo/multirepo is a choice that's very difficult to change once work on building the system starts, and it's a choice you commit at the very beginning (and way before the choice starts to matter) - a perfect recipe for not a mere nudge, but a scaffolding the organization itself will grow around, without even realizing it.

Re: The Ingredients of a Productive Monorepo

#32

Post by ashishb »

jbverschoor wrote: post #24

Is there a way to set permissions on certain directories / force partial clones. Not just a sparse clone.

> there a way to set permissions on certain directories / force partial clones.

No. And that's one reason small startups should separate frontend code into a separate monorepo.

If you would like to hire a contractor for SEO/web developer then give them access to frontend code. Keep the backend code segmented out.

Re: The Ingredients of a Productive Monorepo

#33

Post by jbverschoor »

ashishb wrote: post #32
jbverschoor wrote: post #24

Is there a way to set permissions on certain directories / force partial clones. Not just a sparse clone.

> there a way to set permissions on certain directories / force partial clones. No. And that's one reason small startups should separate frontend code into a separate monorepo. If you would like to hire a contractor for SEO/web developer then give them access to frontend code. Keep the backend code segmented out.

This is exactly my point. I like git, I like monorepos, but I do care about control over access and history.

I use git mainly because everybody knows it, tooling is there, etc.

Re: The Ingredients of a Productive Monorepo

#34

Post by HdS84 »

shawabawa3 wrote: post #19
echelon wrote: post #12

Earlier quoted context omitted.

As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

Hey, do you think Gitlab should do anything except running after the next trend and develop shitty not-solutions for that? Why, that could improve Gitlab. We cannot have that!

Re: The Ingredients of a Productive Monorepo

#35

Post by ashishb »

ianpurton wrote: post #16

I've never worked on a mono repo that has the whole organizations code in it. What are the advantages vs having a mono repo per team?

> What are the advantages vs having a mono repo per team?

If you have two internal services you can change them simultaneously. This is really useful for debugging using git bisect as you always have a code that passes the CI.

I might write a detailed blog about this at some point.

Re: The Ingredients of a Productive Monorepo

#36

Post by jbverschoor »

Pawka wrote: post #27
jbverschoor wrote: post #24

Is there a way to set permissions on certain directories / force partial clones. Not just a sparse clone.

It depends on the VCS you use. I don't know any ways to manage read permissions, such as allowing a person to checkout one directory but not another, though you can do that per branch on git. But there are many ways to manage write permissions - limit the directories to which engineers are allowed to push code. E.g. if you use Git, this can be done with Gitolite, which is a popular hosting server. Gitolite has very f…

It's mostly about read/access permissions. I'd like to stay away from any type of git hook tbh

Re: The Ingredients of a Productive Monorepo

#37

Post by lenkite »

AlotOfReading wrote: post #18
ianpurton wrote: post #16

I've never worked on a mono repo that has the whole organizations code in it. What are the advantages vs having a mono repo per team?

One of the big advantages is visibility. You can be aware of what other people are doing because you can see it. They'll naturally come talk to you (or vice versa) if they discover issues or want to use it. It also makes it much easier to detect breakages/incompatibilities between changes, since the state of the "code universe" is effectively atomic.

Not sure if I get it. If you are using a product like Github Enterprise, you are already quite aware of what other people are doing. You have a lot of visibility, source-code search, etc. If you have a CICD that auto-creates issues you already can detect breakages, incompatibilities, etc.

State of the "code universe" being atomic seems like a single point of failure.

Re: The Ingredients of a Productive Monorepo

#38

Post by dezgeg »

shawabawa3 wrote: post #19
echelon wrote: post #12

Earlier quoted context omitted.

As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

> Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo"

The costs of the infra/build/CI work are of course more visible when there is a dedicated team doing it. If there is no such central team, the cost is just invisibly split between all the teams. In my experience this is more costly overall, due to every team rolling their own thing and requiring them to be jack-of-all-trades in rolling their own infra/build/CI.

> And I don't see how discovery changes significantly from browsing through loads of repositories instead of loads of directories in a repository

If repository permissions aren't set centrally but every team gets to micromanage them, then they usually end up too restrictive and you don't get even read-only access.

Re: The Ingredients of a Productive Monorepo

#39

Post by dezgeg »

friendzis wrote: post #23
shawabawa3 wrote: post #19

Earlier quoted context omitted.

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

Exactly. Take your monorepo, split it into n repos by directory at certain depth from root, write very a rudimentary VCS wrapper script to sync all the repos in tandem and you have already solved a lot of pain points. > E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastructure Bitbucket does this out-of-the box :)

> Take your monorepo, split it into n repos by directory at certain depth from root, write very a rudimentary VCS wrapper script to sync all the repos in tandem and you have already solved a lot of pain points.

Then you lose the capability to atomically make a commit that crosses repoes. I'm not sure if there is any forge that allows that, except Gerrit might with its topics feature (I've not gotten the opportunity to try that).

Re: The Ingredients of a Productive Monorepo

#40

Post by vineyardmike »

phinnaeus wrote: post #22
shawabawa3 wrote: post #19

Earlier quoted context omitted.

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

Great call out. Amazon has an extremely effectively polyrepo setup and it’s a shame there’s no open source analog. Probably because it requires infrastructure outside of the repo software itself. I’ve been toying around with building it myself but it’s a massive project and I don’t have much free time.

The Amazon poly-repo setup is an engineering marvel, and a usability nightmare, and doesn't even solve all the major documented problems of poly-repos. The "version set" idea was probably revolutionary when it was invented, but everyone I know who has ever worked at amazon has casually mentioned that their team has at least one college-hire working 25%+ time on keeping their dependency tree building.
Post reply on HN