Live data from Hacker News

Monorepo: please do

medium.com

131–140 of 166 posts

Re: Monorepo: please do

#131
post #127

Observe how the verb "force" gets used 6 times. Monorepos "force the conversation." You the individual contributor are "forced to deal with the situation" and "forced to see the upfront cost" of breaking contracts. Your team is forced to "look up from their component, and see the perspectives of other teams and consumers." All this forcing people to do things the Right Way ( my way ) is surely part of the pushback ag…

Author here. The Right Way :tm: is situational - there isn't one right answer to things like when and how to make contracts, or how to break them. When I used the term "force", you'll see that I'm usually talking about dialog between people and teams. It's not that it's a single right way to do it. There isn't, and anyone who tells you there is has something to sell you, or is inexperienced enough to not have seen en…

Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.)

Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs to come from the other side of the contract. Eg, What are you depending on? Is it a dependency you want to take on, or are you and the shared code likely to diverge in life? These are top of mind decisions in a polyrepo architecture, but from my experience they're often not even considered in a monorepo. Anything checked in is fair game for reuse. This is why I suspect you may be "forcing" the wrong thing.

For reference I've worked in companies large and small, both monorepo and polyrepo. When I worked on Windows back in the 00's the monorepo tooling (SourceDepot) was quite amazing for the time, but the costs of that sort of coordination were also painfully apparent to everyone.

The place I currently work has a monorepo for desktop software and polyrepos for everything else. It isn't a straight up A/B experiment, but anecdotally the pain is higher and shipping velocity lower in the monorepo half of the world. Most of the monorepo pain is related to CI or other costs of global coordination, the kind of things Matt touches on midway (albeit probably too subtlely). I'd be interested to see your counterarguments to those points as well. Do you need fancy dependency management tooling to make your global CI builds fast and reproducible? Matt argues those end up being equivalent to the kind of dependency tooling that's intrinsic to polyrepo architectures anyway.

Re: Monorepo: please do

#132
post #131

Earlier quoted context omitted.

Author here. The Right Way :tm: is situational - there isn't one right answer to things like when and how to make contracts, or how to break them. When I used the term "force", you'll see that I'm usually talking about dialog between people and teams. It's not that it's a single right way to do it. There isn't, and anyone who tells you there is has something to sell you, or is inexperienced enough to not have seen en…

Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.) Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs…

Disclaimer: it depends. :) Since that's not a good answer at all, I'm going to write the rest of this as if I have the answer, even though I know I do not, because it's deeply situational.

Equal care does need to come from the other side of the contract. Most frequently, I see teams B, C, and D in a polyrepo world do the worst of all worlds: take dependencies liberally, pin them in place, and try to forget about them. Of course, high functioning engineering teams (and cultures) will try and avoid this: they will be thoughtful about dependencies, and they will keep them up to date. In practice, they most frequently do not. This is especially true in the enterprise broadly. When we get it wrong, and take a dependency we wish we hadn't, how do we know? When do we know? What is our recourse? If I depend on code in the monorepo that diverges, I'm more likely to know near to the point of divergence (because of the nature of the system). That means the conversation about how to fix it happens sooner. I'm not interested in avoiding error - that's going to happen. I'm interested in how close to the introduction of the error do we understand it, and how do we communicate about its remediation.

As far as CI and global coordination goes, the cost is high in either direction if the system is distributed, and the solutions are similar in my experience. I think the worst case is the mixed one (which is a world I inhabit) - you wind up splitting your investment in both style and effort across both approaches. With the monorepo style, one big advantage is where the complex CI interactions can be encoded, since you have access to more of the code itself. Granted, at scale, you likely are testing against artifacts rather than point in time commits outside of the component in question (this is very similar to what you're going to do in a polyrepo, too.)

I think solid testing design requires real effort and understanding of the system under test, regardless of repository layout. Which brings us back to communications again. The more you can see, and the more clearly experienced the pain is across the teams, the more likely you are to have the critical conversations needed to improve the system - rather than making local fixes ("my teams tests are fast", "their component sucks").

Re: Monorepo: please do

#133

I wonder if a star pattern would work, where you have a single, shared repo for all your libraries and a repo for each app. This would help people working on smaller apps, since they don't need to look at other apps unless they're working on shared library code. Of course, once you are working on library code, you have to build and test all the apps that use it. But even at Google, the people working on the lowest le…

A star pattern still has most of the downsides of the multirepo approach. Specifically, it has the problem of needing a parallel version control (e.g. SemVer) on top of your individual repositories. This creates fragmentation, where different applications have dependencies on different versions of the libraries which ends up in dependency hell, technical debt, and CI hell.

An alternative would be to have a policy where all the app repos must use the same version (nobody can upgrade until they all upgrade). This makes things harder for the library maintainers, but no more than a monorepo.

I don't see why you'd need semver. The apps could sync to a particular commit in the library repo.

Re: Monorepo: please do

#134

My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck wit…

> Unmigrated semver breaking changes are a form of technical debt Maybe you can clear my confusion. If Module B is dependent on Module A, then every version of B should refer to a specific version of A, correct? What is there to break? Development can continue on A without interfering with B, and then you can uptick B once it points to a later A. I'm not sure what this has to do with the mono/poly discussion.

Engineering resources are not unlimited, so naturally new bugs and features will be updated only on master vs. 2 or 5 major semver branches, because 2-5 module Bs haven't bothered updating yet. If you maintain 5 separate branches, then you're spending that much more engineering resources for little benefit, because you don't have external customers. So the modules that haven't migrated yet decay under a state of deferred maintenance, which is a form of technical debt.

To avoid that, you do 10 migration commits so everyone is on the latest version. If you're going to do that as standard operating procedure anyway might as well make it far easier and have a monorepo.

Re: Monorepo: please do

#135
post #131

Earlier quoted context omitted.

Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.) Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs…

Disclaimer: it depends. :) Since that's not a good answer at all, I'm going to write the rest of this as if I have the answer, even though I know I do not, because it's deeply situational. Equal care does need to come from the other side of the contract. Most frequently, I see teams B, C, and D in a polyrepo world do the worst of all worlds: take dependencies liberally, pin them in place, and try to forget about them…

Most frequently, I see teams B, C, and D in a polyrepo world do the worst of all worlds: take dependencies liberally, pin them in place, and try to forget about them.

This has been my observation as well, minus the value judgment. Why is pinning dependencies and moving on with life the worst thing in the world? As you point out in your article, a security fix in A does suddenly force B, C, and D’s hand. Another scenario I’ll add to that: if A provides communication between B, C and D, a synchronized update to all dependents might be required.

Thing is, I’d argue these scenarios are the exception to the rule. If you’re drawing boundaries in the right places (again this may come back to contract design) you’re largely free to change implementation details when you need to, on your own terms, and not because some distant transitive dependency has decided it’s time for your build to break.

With monorepos I see lots of the latter. Lots of breakage for no other reason than “everyone needs to be on the same page.” Lots of conversations — O(N^2) conversations, times some constant factor — that might not need to take place, ever, but it’s critical the entire company have them right now because the global build is broken.

Here’s another way of looking at it. Until a few years ago, it was standard practice to frequently update npm dependencies against fuzzy semvers. Now most people pin their dependencies, and their dependencies’ dependencies, with a lockfile. And in other ecosystems like go’s you also have tooling to support much more controlled, infrequent and minimal dependency upgrades (see MVS).

Why the change? Because people got tired of things breaking all the time. They wanted off the treadmill so they could Get Things Done again. I don’t see how monorepos provide this stability, and frankly it seems like the monorepo idea is where npm was about 5 years ago. Perhaps even farther behind than that, since C, C++ and others haven’t even evolved viable language package managers yet.

You’re a rust fan, so maybe cargo + a monorepo is a sweet spot I haven’t encountered yet? Anyway, I do really appreciate you taking the time to share your perspective on these things. It’s been great having a reasonable discussion about them.

Re: Monorepo: please do

#136
This is starting to get a debate of "principles", like forcing A and B to talk, or forcing A and B to have more explicit boundaries, and so on. Guess where that ends (hint: it doesn't).

With a monorepo, the basic effort you have to put in to start scaling is quite high. To properly do a local build, you need bazel or something. But bazel doesn't stop at just building, but it manages dependencies all the way down to libraries and stuff. Let's say you're using certain maven plugins, like code coverage, shading, etc. Would bazel have all the build plugins your project needs? Most likely not. You have to backport a bunch of plugins from maven to bazel and so on. Guess how many IDEs support bazel? Not a lot.

Then you need to run a different kind of build farm. When you check-in stuff to a monorepo, you need to split and distribute one single build. Compared to a polyrepo where one build == one job, a monorepo is like one build == a distributed pool of jobs, which again needs very deep integration with the build tool (bazel again here), to fan out, fan in across multiple machines, aggregate artifacts, and so on.

Then the deployment. Same again. There is no "just works" hosted CI or hosted git or anything for monorepos. People still dabble with concourse or so on.

And guess what, for a component in its own repo, you don't need to do anything. Existing industry and OSS tooling is built from ground up for that. Just go and use them.

To provide a developer a "basic experience" to go from working on, building and deploying a single component – the upfront investment you need to provide with a monorepo is very high. Most companies cannot spend time on that, because scale means different things to different companies. There is a vast gap in the amount of ops/dev tooling you have for independent hosted components vs monorepo tools. Just search for "monorepo tools" or DAG and see how many you can come up with. So what really happens with a monorepo is, most companies go with multi-module maven and jenkins multi-job. The results are easy to predict. I'm not saying that maven/jenkins are bad, but they are _not_ sophisticated, and are not anywhere close to what Twitter/Facebook/Google or any modern company uses to deal with a monorepo (for a good reason). They are just not good at DAG. If you're relying on maven+jenkins as your monorepo solution, all I can say is "good luck".

Instead, if you start by putting one component in one repo, you keep scaling for _much longer_ before you hit a barrier.

In principle, monorepos are better. In practice, they don't have the basic "table stakes" tooling that you need to get going. Maybe monorepo devops tooling is a next developer productivity startup space. But until then, it's not mainstream for very good reasons.

Re: Monorepo: please do

#137
post #42
post #33

Earlier quoted context omitted.

But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...

I don't understand. Where is the argument for more productivity?

A: You avoid issues such as Readme files stating, "before compiling you have to git clone ../commonA, ../commonB". These always tend to get stale so in reality you also have to git clone ../commonC wasting you tons of hours of troubleshooting.

B: Developer working on daily basis in component A finds a bug in component B. He just has to change the code and commit it for review, instead of understanding the specifics of working with component B repository.

Re: Monorepo: please do

#138
post #135

Earlier quoted context omitted.

Disclaimer: it depends. :) Since that's not a good answer at all, I'm going to write the rest of this as if I have the answer, even though I know I do not, because it's deeply situational. Equal care does need to come from the other side of the contract. Most frequently, I see teams B, C, and D in a polyrepo world do the worst of all worlds: take dependencies liberally, pin them in place, and try to forget about them…

Most frequently, I see teams B, C, and D in a polyrepo world do the worst of all worlds: take dependencies liberally, pin them in place, and try to forget about them. This has been my observation as well, minus the value judgment. Why is pinning dependencies and moving on with life the worst thing in the world? As you point out in your article, a security fix in A does suddenly force B, C, and D’s hand. Another scena…

If you've got a pre merge build check you can't break global build in a monorepo. That's the benefit, the one introducing the breakage will get a fail in your CI. There is no need for other teams to catch up.

By doing this you only ever "step" a dependency one at a time and one minor minor version at a time, so you only get very few and very small breakages each time. Instead of locking your depfile and then 6 months down the road you realize you need a security fix in component foo but then you got 1000 other backwards incompatible changes to fix because of transitive dependencies that also need to be upgraded in order to satisfy foo 1.2 dependencies.

Re: Monorepo: please do

#139

Earlier quoted context omitted.

A star pattern still has most of the downsides of the multirepo approach. Specifically, it has the problem of needing a parallel version control (e.g. SemVer) on top of your individual repositories. This creates fragmentation, where different applications have dependencies on different versions of the libraries which ends up in dependency hell, technical debt, and CI hell.

An alternative would be to have a policy where all the app repos must use the same version (nobody can upgrade until they all upgrade). This makes things harder for the library maintainers, but no more than a monorepo. I don't see why you'd need semver. The apps could sync to a particular commit in the library repo.

What you propose is just a fake monorepo, containing the global policy of allowed version of X, disguised as multiple subrepos. OP discusses this.

Re: Monorepo: please do

#140

Earlier quoted context omitted.

My old boss was an engineering manager at Google in the 90s and early 2000s. He used to tell us that _everyone_ he interacted with at Google _hated_ the monorepo, and that Google’s in-house tooling did not actually produce anything approaching a sane developer experience. He used to laugh so cynically at stories or that big ACM article touting Google’s use of a monorepo (which was a historical unplanned accident base…

His experience from the 90s and early 2000s is meaningless in the current era. Version control and Google were in their infancy. SVN was first released in 2000. Git in 2008. Branching, tagging and diffing were nowhere near what is possible now. That goes back to desktop with a disk smaller than a GB, CPU in the tens of MHz with a network so slow and reliable, if you have one at all.

My understanding from many Google employees is that the properties of the system that caused problems in ~2000 - 2010 are largely still the same today: the canary node model of deployment, fixed small set of supported languages, code bloat, inability to delete code, bias towards feature toggles even when separate library dependency management would be better for the problem at hand, various firefighting when in-house monorepo tooling breaks, difficult on-boarding for people unfamiliar with that workflow, difficult recruiting for candidates who refuse to join if they have to work under the limits of a monorepo like that.
Post reply on HN