Live data from Hacker News

Monorepo: please do

medium.com

61–70 of 166 posts

Re: Monorepo: please do

#61

Earlier quoted context omitted.

I feel like you've really done no work supporting your argument there. "Slow and inefficient"... what, exactly, is slow and inefficient? Because there are plenty of things slow and inefficient about polyrepos. I'd say that open-source best practices for shared libraries are appropriate if you're making an open-source shared library. However, these practices are inappropriate for internal libraries, proprietary librar…

However, these practices are inappropriate for internal libraries, proprietary libraries, and other use cases. Why do you say so?

Basically because for certain projects and teams, the effort to package internal / proprietary libraries and other similar dependencies can be much larger than the benefit. Packaging is effort. You decide to cut a release, stamp a version number, write a changelog, package and distribute it, and then backport fixes into a long-running branch.

This effort makes a lot of sense if your consumers are complete strangers who work for other organizations. If your consumers are in the same organization, then there are easier ways to achieve similar benefits. See Conway’s Law. It’s not an accident that code structure reflects the structure of the organization that created it, I would claim that organizational boundaries should be reflected in code. Introducing additional boundaries between members of the same organization should not be done lightly.

One of the main benefits of version numbers is that it tells your consumers where the breaking changes are, but if you have direct access to your consumers’ code and can commit changes, review them, and run their CI tests, then you have something much better than version numbers. If you are running different versions of various dependencies you can potentially have a combinatoric explosion of configurations. Then there’s the specter of unknown breaking changes being introduced into libraries. It happens, you can’t avoid it without spending an unreasonable amount of engineering effort, but the monorepo does make the changes easier to detect (because you can more easily run tests on downstream dependencies before committing).

Cross-cutting changes are also much more likely for certain types of projects. These are difficult with polyrepos for obvious reasons (most notably, the fact that you can’t do atomic commits across repos).

Packaging systems also have administrative overhead. If you shove everything in a monorepo you can ditch the packaging system and spend the overhead elsewhere. These days it’s simple enough to shove everything in the same build system.

Various companies that I’ve worked for have experimented with treating internal libraries the same way that public libraries are treated—with releases and version numbers. Most of them abandoned the approach and reallocated the effort elsewhere. The only company that I worked for that continued to use internal versioning and packaging was severely dysfunctional. One startup I worked for went all in on the polyrepo approach and it was a goddamn nightmare of additional effort, even though there were only like three engineers.

Re: Monorepo: please do

#62

Its almost as if both approaches have positives and negatives. Some of which are more important depending on your project and organization. I'd be more interested to read about a project or company that failed due to making one choice or the other. And then by switching things to the other way, things were fixed. Otherwise, as someone who was worked with both, I imagine there are a host of other decisions that will b…

>Its almost as if

Please don't do this.

Re: Monorepo: please do

#63
Can we just move along and get to "Monorepo: Maybe do it, maybe don't. Just think it through and own your decision"?

Both monorepos and polyrepos have advantages and disadvantages. Many factors — scale, overall team quality and experience, level of integration between projects are a few that come to mind — will affect how much those advantages and disadvantages matter to any given company at any given point in time. The right choice for you isn't necessarily the right choice for me.

Much more important than which approach you choose is understanding, and accepting, the consequences of your choice. You'll want to extract value out of the advantages, you'll need to mitigate the disadvantages. You won't be able to adopt tools and processes meant for the other approach without some degree of friction.

Re: Monorepo: please do

#64

Please don't. It's just too slow and not efficient. Instead use common open source best practices of shared library architecture. Problem solved! Putting everything into one repo is just lack of organization and creates a huge mess.

Too slow as in "to do it" or too slow as in "to use it". In either case I think if that were true there wouldn't be monorepo's at Google, Facebook, and Microsoft. I will say it's true that didn't come for free, e.g. Microsoft had to make GVFS due to the sheer enormity of their codebase but that's already done and works pretty well. I agree share library style makes more sense in most cases though. The main problem wi…

Last time I work at a massive Monorepo, half of my team was running got fetch as a cron job. It was an extremely painful experience

Re: Monorepo: please do

#65
post #55

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…

> With 500 it becomes nearly impossible to do anything involving a build. Both FB and Google have more than 500 devs and are using a monorepo.

At what cost? Both FB and Google employ hundreds of devs to work on internal tooling only. For most companies this isn't feasible.

Re: Monorepo: please do

#66

Earlier quoted context omitted.

However, these practices are inappropriate for internal libraries, proprietary libraries, and other use cases. Why do you say so?

Basically because for certain projects and teams, the effort to package internal / proprietary libraries and other similar dependencies can be much larger than the benefit. Packaging is effort. You decide to cut a release, stamp a version number, write a changelog, package and distribute it, and then backport fixes into a long-running branch. This effort makes a lot of sense if your consumers are complete strangers w…

I really don't understand what you describe as effort or huge burden. Writing a simple script that can solve your releasing tasks is simple. Imo a lot of engineers just want to write code but a lot of the time building software consists of other things too, such as testing, releasing, documentation etc. Simply avoiding them doesn't make it better.

Re: Monorepo: please do

#67

Earlier quoted context omitted.

Too slow as in "to do it" or too slow as in "to use it". In either case I think if that were true there wouldn't be monorepo's at Google, Facebook, and Microsoft. I will say it's true that didn't come for free, e.g. Microsoft had to make GVFS due to the sheer enormity of their codebase but that's already done and works pretty well. I agree share library style makes more sense in most cases though. The main problem wi…

I work at one of the monorepo companies that you mention and there’s some truth to the “too slow” part. Although it’s it’s been a lot better lately (largely, due to the efforts of the internal version control dev teams), I’ve noticed at times in the past that you could do a ‘ pull’, go on a 15 minute break and it wouldn’t be done by the time you’re back. Personally, I think there’s a place for mono repos and there’s…

I hate how these monorepo pieces gloss over the CI requirements. Just checkout the code that's affected by the change? Either you have a shared buid job that adds thoussands of builds a day & matching a commit to a build takes ages, or you have a plethora of jobs for each subrepo and Jenkins eats all the disk space with stale workspaces. And let's not talk about how to efficiently clone a large repo... our big repo took 5 minutes to clone from scratch, which killed our target time of 10 minutes from push to test results. We ran git mirrors on our build nodes to have fresh git objects to shallow/reference clone from to get it down to 30 seconds, and the whole system had to work perfectly or else hundreds of devs would be blocked waiting to see if their changes could be merged.

Re: Monorepo: please do

#68

Earlier quoted context omitted.

My org went from a monorepo where every project had to obey the same CI model and you could not introduce entirely new CI tools for new prototypes over to a polyrepo with separated semver library repos for shared dependencies, and it simplified everything so much. Adding additional PRs across different repos is functionally no different than the same PR with scattered dependencies in a monorepo, except that separatin…

This has nothing to do with monorepos though. Its entirely a company policy issue. There's nothing about the monorepo that prevents you from writing a script that ran on precommit and built and deployed via docker to a test cluster. Unless you mean your presubmit test would push to production machines, that's bad and shouldn't be allowed, but again has nothing to do with a monorepo. A company could just as easily hav…

If a tool begets using it wrong all the time, then after a certain point, it’s the tool’s fault.

What you’re saying amounts to something of a No True Scotsman fallacy... “no _real_ monorepo would limit different projects from using individualized tooling if needed...” Yet that limitation suspiciously coexists with monorepo tooling frequently, and does not frequently coexist with polyrepo tooling.

Re: Monorepo: please do

#69

"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it do…

I couldn't agree more!

Re: Monorepo: please do

#70
post #28

Earlier quoted context omitted.

I'm currently working on a monorepo now in our ~30 person engineering organization. We have a microservice architecture, and would like to avoid rebuilding the things that don't need to be rebuilt; however, it seems like most tooling assumes that the whole repo is your project. I've tried working around this by diffing against master or diffing against HEAD~1 or keeping a special tag that tracks the last good state,…

> (and unless I'm mistaken, it seems to assume that all of your code lives in your repository--pulling dependencies from Pypi seems like a fourth class citizen). My feelings here are apart from your tool of choice (Pypi) so read them with that in mind. Why are you dependent on 3rd party code that isn't in your repo? I am a huge advocate of the monorepo and vendoring. Depending on your tooling of choice and your workf…

> Why are you dependent on 3rd party code that isn't in your repo

Not the parent, but for us, the 3rd party code is in a private package manager (artifactory, private npm, whatever). Having thousands of libraries we didn't write in our repo doesn't sound like fun.

Post reply on HN