Monorepo: please do

Its almost as if both approaches have positives and negatives. Some of which are more important depending on your project and organization.

I'd be more interested to read about a project or company that failed due to making one choice or the other. And then by switching things to the other way, things were fixed.

Otherwise, as someone who was worked with both, I imagine there are a host of other decisions that will be much determinant on your success.

Let's not get too wrapped up in what color to paint the shed.

I wonder if a star pattern would work, where you have a single, shared repo for all your libraries and a repo for each app.

This would help people working on smaller apps, since they don't need to look at other apps unless they're working on shared library code.

Of course, once you are working on library code, you have to build and test all the apps that use it. But even at Google, the people working on the lowest levels of the system can't use the standard tools anyway.

My old boss was an engineering manager at Google in the 90s and early 2000s. He used to tell us that _everyone_ he interacted with at Google _hated_ the monorepo, and that Google’s in-house tooling did not actually produce anything approaching a sane developer experience. He used to laugh so cynically at stories or that big ACM article touting Google’s use of a monorepo (which was a historical unplanned accident based on toppling a poorly planned Perforce repository way back when), because in his mind, his experience with monorepos at Google was exactly why his engineering department (several hundred engineers) in my old company did not use a monorepo.

I'm currently working on a monorepo now in our ~30 person engineering organization. We have a microservice architecture, and would like to avoid rebuilding the things that don't need to be rebuilt; however, it seems like most tooling assumes that the whole repo is your project. I've tried working around this by diffing against master or diffing against HEAD~1 or keeping a special tag that tracks the last good state, and all of these seem to spawn odd edge cases and generally are tedious when your build target is entirely contained in a single directory--no idea how to solve for build targets that depend on files elsewhere in the repo. A tool like Bazel seems really heavy for an organization our size (and unless I'm mistaken, it seems to assume that all of your code lives in your repository--pulling dependencies from Pypi seems like a fourth class citizen).

Have you (or anyone reading this thread) encountered similar issues? How do you solve them in a monorepo?

I have personally migrated a medium size polyrepo code base (something like ~20 repos?) into a monorepo and I agonized over the decision. But it lifted a huge weight off my shoulders.

I feel like if you are working completely in the open-source world, and you are contributing one open-source project to a larger array of available projects, then the decision to use a polyrepo makes a lot of sense. You can submit libraries to a package repository like Yarn/NPM/PyPI or you can use Git references for e.g. Go's package manager.

But what I experienced with polyrepos outside this world is that we ended up with a weird DAG of repos. It was always unclear whether a specific piece of code that was duplicated between projects should be moved into one dependency or another, or whether it should have its own repo. Transitive dependencies were no fun at all, if you used git modules you might end up with two copies of the same dependency. You might have to make a sequence of commits to different repos, remembering to update cross-repo references as you go, and if you got stuck somewhere you had to work backwards. This feels like a step backwards, like the step backwards from CVS to RCS.

Again, in the open-source world you might have some of this taken care of by using a package manager like Yarn. But if your transitive dependencies aren't suitable for being published that way, it can be tough. Monorepo + Bazel right now is a bit rough around the edges but overall it's reduced the amount of engineering time spent on build systems.

On the other hand, it's not like Bazel can't handle polyrepos. In fact, they work quite nicely, and Bazel can automatically do partial checkouts of sets of related polyrepos, if that's your thing.

As for VCS scalability problems, I expect that Git is really just the popular VCS du jour and some white horse will show up any day now with a good story for large, centralized repos with a VFS layer. In the meantime any company large enough to experience VCS performance problems but not large enough have their own VCS team (like Google and Facebook) will suffer, or possibly pay for Perforce.