[deleted]
The Ingredients of a Productive Monorepo
61–70 of 268 posts
Re: The Ingredients of a Productive Monorepo
#62This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo.
This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration.
To me the whole point of a monorepo is atomic commits for the whole org. The power of this is really hard to overstate when you are trying to orchestrate the efforts of lots of developers - contrary to many claims. Rebasing in one repo and having one big meeting is a hell of a lot easier than doing it N times.
Even if the people on the team hate each other and refuse to directly collaborate. I still don't see the reason to not monorepo. In this scenario, the monorepo becomes a useful management and HR tool.
Re: The Ingredients of a Productive Monorepo
#63Genuine question, because I've never worked somewhere with a monorepo infrastructure: is it really "one repo for all code in the organization" or "one repo for everything related"? In my organization we have around 70k internal git repos (and an order of magnitude fewer public ones), but of course not everything is related to everything else; we produce many distinct software products. I can understand "collect every…
In game dev monorepo per product is often used, which includes game code, art assets, build system and tooling, as well as engine code that can receive project-specific patches. In Perforce, it's organised into streams, where development streams are regularly promoted to staging, then to release, etc.
The benefit is the tooling, as the article mentioned. Everything in the repo is organised consistently, so I can make ad-hoc Python tools relying on relative paths knowing that my teammates have identical folder structure.
Re: The Ingredients of a Productive Monorepo
#64Earlier quoted context omitted.
It's worth noting that most monorepos won't reach the same size as repositories from Google, Uber, or other tech giants. Some companies introduce new services every day, but for some, the number of services remains steady. If a company has up to 100 services, there won't be VCS scale problems, LSP will be able to fit the tags of the entire codebase in a laptop's memory, and it is probably _almost_ fine to run all tes…
I do think the 'run all tests on CI' part is not that fine, it bites a lot earlier than the others do. Git is totally fine for a few hundred engineers and 100ish services (assuming nobody does anything really bad to it, but then it fails for 10 engineers anyway), but running all tests rapidly becomes an issue even with tens of engineers. That is mitigated a lot by a really good caching system (and even more by full r…
There are also so many types of slow tests in web systems. Any kind of e2e test like Cypress or Playwright can easily take a minute. Integrations tests that render components and potentially even access a DB take many times longer than a basic unit test. It doesn’t take very many of the slow group to reaaaly start slowing your system down. At that point, what matters is how much money you’re willing to pay to scale your build agents either vertical or (more likely) horizontally
Re: The Ingredients of a Productive Monorepo
#65An unspoken truth of a monorepo is that everyone is committed to developing on trunk, and trunk is never allowed to be broken. The consequence of this is that execution must be configurable at runtime: feature flags and configuration options with old and new code alongside each other.
You can have a monorepo and still fail if every team works on their own branch and then attempts to integrate into trunk the week before your quarterly release process begins.
You can fail if a core team builds a brand new version of the product on master with all new tests such that everything is green on every commit but your code is unreleasable because customers aren’t ready for v2 and you need to keep that v1 compatability around.
Re: The Ingredients of a Productive Monorepo
#66Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?
The whole point of a monorepo is to force you to update all of the consumers, and to realize that breaking changes are expensive. The two monorepo ways to do this: 1. Use automated refactoring tools that now work because it's one repo 2. Add the new behavior, migrate incrementally, then remove the old behavior
Thanks for the info!
Seems like a big restriction to me.
Re: The Ingredients of a Productive Monorepo
#67Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?
That's the neat part. They don't. Either the broken consumer updates their use, you update it for them to get your change shipped, or you add some backwards compatibility approach so your breaking changes aren't breaking.
Thanks for the info!
Seems like a big restriction to me.
Re: The Ingredients of a Productive Monorepo
#68> Meta has a sophisticated implementation of a target determinator on top of buck2, but I don’t believe it is open-source.
It is: https://github.com/facebookincubator/buck2-change-detector
> Some tools such as bazel and buck2 discourage you from checking in generated code and instead run the code generator as part of the build. A downside of this approach is that IDE tools will be unable to resolve any code references to these generated files, since you have to perform a build for them to be generated at all in the first place
Not an issue I have experienced. It's pretty difficult to get into a situation where your IDE is looking in buck-out/v2/gen/781c3091ee3/... for something but not finding it, because the only way it knows about those paths is by the build system building them. Seeing this issue would have to involve stale caches in still-running IDE after cleaning the output folder, which is a problem any size repo can have. In general, if an IDE can index generated code with the language's own build system, then it's not a stretch to have it index generated code from another one.
The problem is more hooking up IDEs to use your build system in the first place. It's a real slog to support many IDEs.
Buck recently introduced an MSBuild project generator where all build commands shell out to buck2. I have seen references to an Xcode one as well, I think there's something there for Android as well. The rust-analyzer support works pretty well but I do run a fork of it. This is just a few. There is a need (somewhat like LSP, but not quite) for a degree of standardization. There is a cambrian explosion of different build systems and each company that maintains one of them only uses one or two IDEs and integrates with those. If you want to use a build system for an IDE they don't support, you are going to have a tough time. Last I checked the best effort by a Language Server implementation at being build-system agnostic is gopls with its "gopackagesdriver" protocol, but even then I don't think anyone but Bazel has integrated with it: https://github.com/bazel-contrib/rules_go/wiki/Editor-and-to...
Re: The Ingredients of a Productive Monorepo
#69Genuine question, because I've never worked somewhere with a monorepo infrastructure: is it really "one repo for all code in the organization" or "one repo for everything related"? In my organization we have around 70k internal git repos (and an order of magnitude fewer public ones), but of course not everything is related to everything else; we produce many distinct software products. I can understand "collect every…
When you have N repos, you also have N ways of managing dependencies, N ways of doing local bin scripts and dev environment setups, N projects with various out of date & deprecated setups, N places to look when you need to upgrade a vulnerable dependency, N services which may or may not configure telemetry in a consistent way, N different CI & deployment workflows…
It just gets very difficult to manage, especially if people frequently need to work across many repos. Plus, onboarding is a pain in the ass.
Monorepo example: if I want to add a new Typescript package/library for internal NodeJS use, we have a boot strapping script that sets it up. And it basically:
1. Inherits a tsconfig that just works in the context of the repo
2. Jest is configured with our default config for node projects and works with TS out of the box.
3. Listing / formatting etc are all working out of the box.
4. Can essentially use existing dependencies the monorepo uses
5. Imports in existing code work immediately since it’s not an external dependency
6. CI picks up on the new typescript & jest configs and adds jobs for them automatically
7. Code review & collaboration is happening in the same spot
8. This also makes it easier to have devs managing the repo — for example, routine work like updating NodeJS is a lot easier when you know everything is using a nearly identical setup & is automatically verified in CI.
One challenge I had to help solve in a previous job was that onboarding was difficult because we had a small number of large repos everyone worked in. The standards were slightly different across them. Npm, pnpm, and yarn were all in use. Deployment worked pretty differently among them. CI setups were unique, and each of the large projects had, if not a team, some number of people spending a lot of time just managing the project’s workflows.
So many coordination things just get easier when there isn’t an opportunity to get out of sync. If you do separate repos, you can totally share config… but now it costs a dependency update PR to pull in that tiny update to the shared unit test config and now everything. It’s just guaranteed to get out of sync, and it’s hard to catch issues when you can’t validate a config change with all projects using it at the same time.
So because it becomes trickier (and takes work) just to do the action of syncing multiple repo’s setups… inevitably, you end up with some “standards” that are loosely followed and a lot of slightly different setups that get hard to untangle the longer they grow. If you can accept the cost of context switching between repos, or if people don’t need to switch, maybe it’s ok… until something like a foundational dependency update (NodeJS, Typescript, React, something like that) needed for security becomes extremely difficult because you have a million different ways of configuring things and the JS ecosystem sucks
Re: The Ingredients of a Productive Monorepo
#70Earlier quoted context omitted.
> force you to update all of the consumers, and to realize that breaking changes are expensive. ...and the article points out correctly that it's a lie anyway, but at least you can find all the consumers easily.
The article is not correct on that point. At Google we would create release branches to fix the monorepo at a predictable point for testing, and only cherry-pick what we need during the release process. I'm sure others do similarly, because there is no way you would allow arbitrary changes to creep in the middle of a multi-service rollout.
The multi-service staggered rollout is the reason the article is correct unless you are tolerating contract mismatches somehow other than in the code. not at google so won't be guessing.