51–60 of 268 posts
This is a good thought! It actually needs to be O(1/commit rate) though, so that having the monorepo doesn't create long queues of commits.
Or have some process batch passing ready to merge PRs into a combined PR and try to merge that. And best guess on the failing PR if it fails.
Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?
So there are 2 kinds of big tech monorepos. One is the kind described in the article here: "THE" monorepo of the (mostly) entire codebase, requiring custom VCS, custom CI, and a team of 200 engineering supporting this whole thing. Uber and Meta and I guess Google do it this way now. It takes years of pain to reach to this point. It usually starts with the other kind of "monorepo": The other kind is the "multirepo mon…
As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…
Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?
The two monorepo ways to do this:
1. Use automated refactoring tools that now work because it's one repo
2. Add the new behavior, migrate incrementally, then remove the old behavior
Wireit is the smallest change from plain npm that gets you a real dependency graph of scripts, caching (with GitHub Actions support), incremental script running, and services.
Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?
The whole point of a monorepo is to force you to update all of the consumers, and to realize that breaking changes are expensive. The two monorepo ways to do this: 1. Use automated refactoring tools that now work because it's one repo 2. Add the new behavior, migrate incrementally, then remove the old behavior
...and the article points out correctly that it's a lie anyway, but at least you can find all the consumers easily.
This issue is that users of a library can put almost infinite friction on the library. If the library team wants to make a change, they have to update all the use sites, but Hyrum's Law will get you because users will do the damndest things.
So for the top organization, it's good if many other teams can utilize a great team's battle-tested library, but for the library team it's just liability (unless making common code is their job). In a place like Google you either end up with internal copies and forks, strict access control lists, or libraries that are slow as molasses to change.
In my organization we have around 70k internal git repos (and an order of magnitude fewer public ones), but of course not everything is related to everything else; we produce many distinct software products. I can understand "collect everything of a product to a single repo"; I can even understand going to "if there is a function call, that code has to be in the same repo". But putting everything into a single place... What are the benefits?
Earlier quoted context omitted.
The whole point of a monorepo is to force you to update all of the consumers, and to realize that breaking changes are expensive. The two monorepo ways to do this: 1. Use automated refactoring tools that now work because it's one repo 2. Add the new behavior, migrate incrementally, then remove the old behavior
> force you to update all of the consumers, and to realize that breaking changes are expensive. ...and the article points out correctly that it's a lie anyway, but at least you can find all the consumers easily.
I'm sure others do similarly, because there is no way you would allow arbitrary changes to creep in the middle of a multi-service rollout.
Earlier quoted context omitted.
Great call out. Amazon has an extremely effectively polyrepo setup and it’s a shame there’s no open source analog. Probably because it requires infrastructure outside of the repo software itself. I’ve been toying around with building it myself but it’s a massive project and I don’t have much free time.
The Amazon poly-repo setup is an engineering marvel, and a usability nightmare, and doesn't even solve all the major documented problems of poly-repos. The "version set" idea was probably revolutionary when it was invented, but everyone I know who has ever worked at amazon has casually mentioned that their team has at least one college-hire working 25%+ time on keeping their dependency tree building.