Live data from Hacker News

Monorepos: Please don’t

medium.com

391–400 of 402 posts

Re: Monorepos: Please don’t

#391

Earlier quoted context omitted.

In my experience, monorepos cause outrageous problems that have nothing to do with scale. Small or medium monorepos are equally as terrifying. It’s much more about coupling and engendering reliance on pre-existing CI constraints, pipeline constraints, etc. If you work in a monorepo set up to assume a certain model of CI and delivery, but you need to innovate a new project that requires a totally different way to appr…

Absolutely amazing to me how much engineers conflate organizational issues with tooling issues. Let's take a look at one of your comments: "The last point is not trivial. Lots of people glibly assume you can create monorepo solutions where arbitrary new projects inside the monorepo can be free to use whatever resource provisioning strategy or language or tooling or whatever, but in reality this not true, both because…

Nobody is conflating anything. Culture / sociological issues that happen to frequently co-occur with technology X are valid criticisms of technology X and reasons to avoid it.

To argue otherwise, and draw attention away from the real source of the policy problems (that the monorepo enables the problems) is a bigger problem. It’s definitely some variant of a No True Scotsman fallacy: “no _real_ monorepo implementation would have problems like A, B, C...”.

The practical matter is that where monorepos exist, monopolicies and draconian limitations soon follow. It’s not due to some first principles philosophical property of monorepos vs polyrepos — who cares! — but it’s still just the pragmatic result.

Also you mention,

> “Where was the conversation about whether GPU accelerated ML jobs were worth the additional business value to change the deployment pipeline.”

but this was explicitly part of the product roadmap, where my team submitted budgets for the GPU machines, we used known latency and throughput specs both from internal traffic data and other reference implementations of similar live ML models. Budgeting and planning to know that it was cost effective to run on GPU nodes was done way in advance.

The people responsible for killing the project actually did not raise any concern about the cost at all (and in fact they did not have enough expertise in the area of deploying neural network models to be able to say anything about the relative merit of our design or deployment plan).

Instead the decision was purely a policy decision: the code in the monorepo that was used for serving compute tasks just as a matter of policy was not allowed to change to accommodate new ways of doing things. The manager of that team compared it with having language limitations in a monorepo. In his mind, “wanting to deploy using custom Docker containers” was like saying “I don’t want to use a supported language for my next project.”

This type of innovation-killing monopolicy is very unique to monorepos.

Re: Monorepos: Please don’t

#392
post #356

Earlier quoted context omitted.

"To characterize that scale as common is one of the most embarrassing failures of modern software engineering." This point cannot be stressed enough. Almost all the worst software engineering failures I have seen have been caused by premature scaling - which is way worse than premature optimization because the latter's effects are usually local. But premature scaling causes architectural decisions that affects the wh…

This complicated failover setup took huge amount of time and resources to setup, delaying the project by months. Hmm - failover shouldn't be that hard to set up. If it was then that suggests that other issues (technical debt, inexperienced management) were the more likely culprits. Not the simple fact that they chose not to ignore the need for failover.

> [it] shouldn't be that hard ...

Now where have I heard those words before... :)

Re: Monorepos: Please don’t

#393
post #317

Earlier quoted context omitted.

> So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy? If you're using Git, then typically for each push to the remote repository you get a notification with this data in it: BRANCH # the remote branch getting updated OLD_COMMIT # the commit the branch ref was pointing to before the push NEW_COMMIT # the commit the branch ref was pointing to after the push # To g…

This is something I've been working on a bit myself. Figuring out which files change is relatively easy (as you've demonstrated). Figuring out what the impact of that is quite hard in non-compiled languages (tools like Maven, Buck, Bazel, etc do this well for compiled languages). I.e. In a repo which is primarily JavaScript, I can get the list of changed files, and hopefully have unit test files which are obviously l…

I believe the typical approach is to have project.yml list its dependency projects. Build a DAG(error on cycles) and then build all changed and downstream projects.

Re: Monorepos: Please don’t

#394

Maybe its just that the author's cutoff is at the wrong team size, but the monorepo I work on (with ~150 devs) has almost none of the problems presented. Unreasonable for a single dev to have the entire repo? I'm looking at a repo with ~10 million LoC and ~1.4 million commits. I have 74 different branches checked out right now. Hard drives are cheap . Code refactors are impossible? I reviewed two of those this mornin…

In theory, yes - if the underlying repo changes, code review should start over. In practice though, it's a terrible idea ;) Part of code review is to ensure the code "fits" with all other merged code - so a re-review is "needed" when other changes merge. E.g. if I merge a refactor that changes everything from Pascal case to 100% SHOUTING, reviews now need to take this into account. In practice, this doesn't happen -…

I think the trick is to only re-review the areas that had merge conflicts, and to do the re-review aware of both the changes you already reviewed and the changes that caused the conflict. Merge conflicts, even in big code refactors, are fairly rare, so this ends up not being much additional work in practice.

Re: Monorepos: Please don’t

#395
post #191

Earlier quoted context omitted.

>It makes absolutely sense to have a repository for the gui and one for the server. Not really. You can have a single repo with top level directories tigershark-gui and tigershark-server.

And later you add the iOS and Android clients too. Will those go into the same repo? Better to keep server and clients apart, especially if release schedules are different.

Sure if the release schedules are different then have them in separate repos so things like tagging makes sense. But often people work with a single release schedule. There's just so many variables that go into these decisions that the thread here is bonkers.

Smart people can work through problems to get the job done. Monorepo vs polyrepo won't stop people from moving forward.

Re: Monorepos: Please don’t

#396

Earlier quoted context omitted.

> after it's been solved I think a quick perusal of this page will show that it's not really "solved" after all. A far higher percentage of developers continue to be affected by large-repo issues than a Python-specific issue (currently #1 story on the front page) or anything to do with Ethereum (currently #7). Are those "horseshit" topics too?

Incidentally, I also think those are horseshit topics as well (Coconut is someone trying to daydream Python into Haskell with no practical reasons to do so and making Ethereum scale better doesn't make a legitimate use case for it emerge) but that's besides the point. What you call large-repo issues I call organization issues. From your other comments, it's clear that we draw the lines at different places, but I thin…

> Why are there busybodies and style pedants working in your organization?

Because to an extent they serve a useful purpose. In a truly large development organization - thousands of developers working on many millions of lines of code - fragmentation across languages, libraries, tools, and versions of everything does start to become a real problem with real costs. You do need someone to weed the garden, to work toward counteracting that natural proliferation. That improves reuse, economies of scale, smoothness of interactions between teams, ease of people moving between teams, etc. It's a good thing. Unfortunately...

(1) That role tends to attract the very worst kind of "I always know better than you" pedants and scolds. Hi, JM and YF!

(2) Once that team reaches critical mass, they forget that the dog (everyone else) is supposed to wag the tail (them) instead of the other way around.

At this point, Team Busybody starts to take over and treat all code as their own. Their role naturally gives them an outsize say in things like repository structures, and they use that to make decisions that benefit them even if they're at others' and the company's expense. Like monorepos. It's convenient for them, and so it happens, but that doesn't mean it's really a good idea.

Sure, it's a culture issue. So are the factors that lead to the failure of communism. But they're culture issues that are tied to human nature and that inevitably appear at scale. I know it's hard for people who have never worked at that scale to appreciate that inevitability, but that doesn't make it less real or less worth counteracting. One of the ways we do that is by putting structural barriers in the corporate politicians' way, to maintain developers' autonomy against constant encroachment. The only horseshit here is the belief that someone who rode a horse once knows how to command a cavalry regiment.

Re: Monorepos: Please don’t

#397

I think this article is complete horseshit. A monorepo will serve you 99% of the time until you hit a certain level of scale when you get to worry about whether a monorepo or a polyrepo is actually material. Most cases are never going to get there. Before that point, a polyrepo is purely a distraction and makes synchronous deployment really painful. We had to migrate a polyrepo to a monorepo and it was not fun becaus…

> A monorepo will serve you 99% of the time until you hit a certain level of scale when you get to worry about whether a monorepo or a polyrepo is actually material

If you worked in a company that had a core product in a repo, and you wanted to create a slack bot for internal use, where would you put the code? I assume not within your core product's codebase, but within a separate repo, thus creating a polyrepo situation.

So when you say a monorepo will serve you in 99% of cases, are you not counting "side" projects, and simply talking about the core product?

Re: Monorepos: Please don’t

#398
post #113

Earlier quoted context omitted.

My rule of thumb is: if you need to do PRs in several repositories to do one features, you should probably merge the repositories. At work, we have code spread among a bunch of repositories, and having to link to the 2/3 related PRs in other repos is a major PITA, and even more so for the reviewers.

My rule of thumb is: if you need to do PRs in several repositories to do one feature, your projects are either tightly coupled enough that they should be one monolithic piece of software , or your tight coupling is a problem you should work on resolving. Requiring multiple PRs to multiple repos to roll out one user-facing feature is fine, as long as your independent modules/projects are not actually interdependent (i…

Sometimes a feature needs to change a shared dependency library.

But in that case you could consider the change to the dependency a single release. And ingesting it into another app a separate release.

Re: Monorepos: Please don’t

#399

Earlier quoted context omitted.

“You'd need a way to run all tests in all repos reflecting a new change.” You really shouldn’t have to run every test on every product. Or really any other repos. Use semantic versioning, pin your dependencies, don’t make breaking changes on patch or minor versions.

Pinning your dependencies is an antipattern (or at least in the eyes of many people who support monorepos it is). It results in one of three things: 1. People never update their dependencies. This is bad (consider a security issue in a dependency) 2. Client teams are forced to take on the work of updating due to breaking changes in their dependencies. If they don't, we're back at 1. 3. Library teams are forced to bac…

At AMZN, which has 1000s of separate repos, 1) was the general case, with 2) occurring whenever there was a critical security issue in some library that no one had updated for years. The resulting fire drill of beating transitive dependencies into submission could occupy days or weeks of dev time.

Re: Monorepos: Please don’t

#400
post #113

Earlier quoted context omitted.

My rule of thumb is: if you need to do PRs in several repositories to do one features, you should probably merge the repositories. At work, we have code spread among a bunch of repositories, and having to link to the 2/3 related PRs in other repos is a major PITA, and even more so for the reviewers.

At a past job, I had to edit roughly 5 different repositories in order to do some trivial programming task (send an email or some such). It was quite easily the least productive / most demoralizing workflow I've ever experienced. Context switching really sucks. You should aim to reasonably avoid it

Don't mix up downsides of multirepo and bad composition of your microservices
Post reply on HN