Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

21–30 of 268 posts

Re: The Ingredients of a Productive Monorepo

#21
post #3

This definitely tracks with my experience in big tech - managing large scale build systems ends up taking a team that works on the build system itself. The underlying repo technology itself needs to work at scale & that was with a virtual file system that downloaded source files on demand when you needed to access them. One thing that this article didn't mention is that most development was done either on your develo…

> One thing that this article didn't mention is that most development was done either on your development server running a datacenter (think ~50-100 cores)

I have done this for many small teams as well.

It remains pretty hard to get engineers to stop "thinking localy" when doing development. And with what modern hardware looks like (in terms of cost and density) it makes a lot of sense to find a rack some where for your dev team... It's easy enough to build a few boxes that can run dev, staging, test and what ever other on demand tooling you need with room to grow.

When you're close to your infrastructure and it looks that much like production, when you have to share the same playground the code inside a monorepo starts to look very different.

> managing large scale build systems ends up taking a team that works on the build system itself

This is what stops a lot of small teams from moving to monorepo. The thing is, your 10-20 person shop is never going to be google or fb or ms. They will never have large build system problems. Maintaining all of it MIGHT be someone's part time job IF you have a 20 person team and a very complex product. Even that would be pushing it.

Re: The Ingredients of a Productive Monorepo

#22
post #12

Earlier quoted context omitted.

As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

Great call out. Amazon has an extremely effectively polyrepo setup and it’s a shame there’s no open source analog. Probably because it requires infrastructure outside of the repo software itself. I’ve been toying around with building it myself but it’s a massive project and I don’t have much free time.

Re: The Ingredients of a Productive Monorepo

#23
post #12

Earlier quoted context omitted.

As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

Exactly. Take your monorepo, split it into n repos by directory at certain depth from root, write very a rudimentary VCS wrapper script to sync all the repos in tandem and you have already solved a lot of pain points.

> E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastructure

Bitbucket does this out-of-the box :)

Re: The Ingredients of a Productive Monorepo

#25

Is there a way to set permissions on certain directories / force partial clones. Not just a sparse clone.

You can set permissions on writes.

Optional, per-directory OWNERS files are common, and most VCS frontends (Github, Bitbucket, etc.) can be configured to prevent merges without approval from the owning team(s) or DRI(s).

PRs that intersect multiple teams' ownership would require handoff of everyone impacted. So a team updating the company-wide "requests library" (or an equivalent change), with a wide blast radius, would be notifying everyone impacted and getting their buy-in.

Re: The Ingredients of a Productive Monorepo

#26

One thing I don't usually see discussed in monorepo vs multi repo discussions is there's an inverse Conway's law that happens: choosing one or the other will affect the structure of your organization and the way it solves problems. Monorepos tend to invite individual heroics among common infrastructure teams, for example. Because there are so many changes going in at once, anything touching a common area has a huge n…

Is your underlying assumption that the organization doesn't want to go one way or the other in the first place and is nudged by the technical choice afterwards ?

I think most of the time the philosophical decision (more shared parts or better separation) is made before deciding how you'll deal with the repos.

Now, if an org changes direction mid-way, the handling of the code can still be adapted without fundamentally switching the repo structure. Many orgs are multi-repo but their engineers have access to almost all of the code, and monorepo teams can still have strong isolation of what they're doing, up to having different CI rules and deployment management.

Re: The Ingredients of a Productive Monorepo

#27

Is there a way to set permissions on certain directories / force partial clones. Not just a sparse clone.

It depends on the VCS you use. I don't know any ways to manage read permissions, such as allowing a person to checkout one directory but not another, though you can do that per branch on git.

But there are many ways to manage write permissions - limit the directories to which engineers are allowed to push code. E.g. if you use Git, this can be done with Gitolite, which is a popular hosting server.

Gitolite has very flexible hooks support, especially with so-called "Virtual Refs" (or VREFs)[1]. It is out of the box and has support to manage write permissions per write path [2]. You can go even further and use your own custom binary for VREF to "decide" if a user is allowed to push certain changes. One possible option - read incoming changed files, read metainformation from the repository itself (e.g., CODEOWNERS file at the root of the repo), and decide if push should be accepted. GitHub has CODEOWNERS [3], which behaves similarly.

[1]: https://gitolite.com/gitolite/cookbook.html#vrefs [2]: https://gitolite.com/gitolite/vref.html#quick-introexample [3]: https://docs.github.com/en/repositories/managing-your-reposi...

Re: The Ingredients of a Productive Monorepo

#28
I wrote a bit about monorepo tooling in this blog post. It covers many of the same points in the OP, but in a lot more detail.

- https://mill-build.org/blog/2-monorepo-build-tool.html

People like to rave about Monorepos, and they are great if set up correctly, but there's a lot of intricacies that often goes on behind the scenes to make a Monorepo successful that it's easy to overlook since usually some "other" team (devops teams, devtools team, etc.) is shouldering all that burden. Still worth it, but most be approached with caution

Re: The Ingredients of a Productive Monorepo

#29
The artikel doesn’t bringa it up, but I’ve seen several places where repos has been cut according to company silos, where applikation code was in a monorepo for all teams, IaC was in one monorepo for all teams, and ops was in one monorepo for all teams. It was not good at all.

Re: The Ingredients of a Productive Monorepo

#30
I've been very happy with nix. I've been using nix in the reciperium.com monorepo, granted, it's only me, but I'm quite happy with having everything there. From docs, to the infra with terraform, to frontend and backend. The procedure for the CI is quite straightforward (nix build .#project), and caching the dependencies in the CI works quite okay. Even the secrets are there, encrypted using age (might not be the best, but good enough).
Post reply on HN