Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepo: please do

medium.com

Post reply on HN
Print view View original

41–50 of 166 posts

Re: Monorepo: please do

#41

Post by klodolph »

nkozyra wrote: post #12
erulabs wrote: post #6

Earlier quoted context omitted.

If a mono-repo has a terabyte of code, or if 10 small repos have 1/10th a terabyte each, what have you really gained? In any case, git LFS solves large file storage effectively, as do a number of other artifact storage solutions, and a repo with a terabyte of code is _not_ going to be trivially split apart, since it would be by a factor of thousands, the biggest codebase ever created by humankind.

> If a mono-repo has a terabyte of code, or if 10 small repos have 1/10th a terabyte each, what have you really gained? If it's a small company where every developer touches every part of the application, sure. Taking the FAANG approach if you're not part of that acronym sounds like introducing inefficiency.

If it's a "small" company then I'd expect that one Git repo would do just fine for all or at least most of the code. When I think small, I think ~10 or 20 developers. If you have reasonable hygiene about things like keeping binaries out of your Git repo (excluding consideration of e.g. LFS here) then the whole repo size will stay fairly reasonable. As long as you have one or two Git mavens on your team it should be dandy.

I'd expect to see problems with this approach once you get into the 100s or 1000s of developers. The tooling for this scale of repository isn't as mature.

Re: Monorepo: please do

#42

Post by nkozyra »

pvorb wrote: post #33
nkozyra wrote: post #8

Earlier quoted context omitted.

Huh? You'd download dependencies for the repos you need, not the code and dependencies for the entire company. It could be several orders of magnitude larger and with a larger organization could be a lot of unnecessary code that any given Dev may never touch.

But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...

I don't understand. Where is the argument for more productivity?

Re: Monorepo: please do

#43

Post by [deleted] »

kokokokoko wrote: post #22

Its almost as if both approaches have positives and negatives. Some of which are more important depending on your project and organization. I'd be more interested to read about a project or company that failed due to making one choice or the other. And then by switching things to the other way, things were fixed. Otherwise, as someone who was worked with both, I imagine there are a host of other decisions that will b…

[deleted]

Re: Monorepo: please do

#44

Post by mindcrime »

"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible".

More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it doesn't make sense to have just anybody randomly making changes in parts of the code-base they don't normally work in.

I'd rather see a poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR. Basically "internal open source" or "inner source"[1].

In my experience, this is about as close as you can get to a "best of both worlds" situation. But, as the author of TFA also says, you absolutely can make either approach work.

[1]:https://en.wikipedia.org/wiki/Inner_source

Re: Monorepo: please do

#45

Post by denimnerd42 »

thurn wrote: post #40
totallysnowman wrote: post #19

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…

the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...

yeah by writing custom version control software. Am I going to convince my company to do that (which has like 50k software engineers) probably not.

Re: Monorepo: please do

#47

Post by zer00eyz »

weberc2 wrote: post #28
mmmeff wrote: post #15

Thank you so much for writing this. As someone who’s worked in the best and worst of these two words, the productivity gains are absolutely insane and the limitations, as stated by the author, are no more painful than limitations of federated/polyrepo code. Fighting back against monorepo design is dangerous - embrace experimentation.

I'm currently working on a monorepo now in our ~30 person engineering organization. We have a microservice architecture, and would like to avoid rebuilding the things that don't need to be rebuilt; however, it seems like most tooling assumes that the whole repo is your project. I've tried working around this by diffing against master or diffing against HEAD~1 or keeping a special tag that tracks the last good state,…

> (and unless I'm mistaken, it seems to assume that all of your code lives in your repository--pulling dependencies from Pypi seems like a fourth class citizen).

My feelings here are apart from your tool of choice (Pypi) so read them with that in mind.

Why are you dependent on 3rd party code that isn't in your repo? I am a huge advocate of the monorepo and vendoring. Depending on your tooling of choice and your workflow checks for updates on this third party code should be frequent (security) and done by someone qualified (not a job for the "new guy").

The question is where should this start and end? The answer (for me) is everything and I have elected to use less (and reduce complexity) to avoid bloat. Really though this is an artifact of my use of Git: https://unix.stackexchange.com/questions/233327/is-it-possib... --

Re: Monorepo: please do

#48

Post by mwkaufma »

hinkley wrote: post #37
mwkaufma wrote: post #25

Earlier quoted context omitted.

We have a perforce monorepo with ~80gb total payload for the whole thing, but everyone uses streams to filter it, so that's not a problem.

I think there's a false dichotomy here. In the post yesterday one of the arguments was that if nobody checks out all of the code then what's the value of having the code all in one place? Last monorepo I worked on, individual contributors checked out just the tree they were working on (we had a suite of applications with several shared modules). We made it simple and straightforward for them to get what they wanted a…

Streams aren't modules -- they're views. If someone takes you as a dependency and wants you to have visibility on them they add themselves to your stream so you pull down their directory as well.

Re: Monorepo: please do

#49

Post by lstamour »

hinkley wrote: post #34
sharpercoder wrote: post #2

In my experience, this discussion gets convoluted by confusing modularity with monorepo . They are orthogonal to each other; you can have a very modular codebase in a monorepo but also a very coupled (non-modular) codebase with polyrepo. Though it's true that monorepos without proper discipline can tend towards coupling. Yet, when discussing mono vs poly, we should keep this in mind.

First monorepo I worked on, we used separate compilation units for each 'module'. We paid a tax on build time but it added a bit of friction to adding new cross-module dependencies willy nilly. I don't know how you maintain that arm's length separation if you don't have compilation units in your language of choice, and that may contribute to some of the muddiness in this kind of discussion. "It depends."

I think the private visibility and shared build chain that Bazel offers could step in here, in that it makes it harder to build a project without specifying every dependency, when combined with linting tools and clearly assigning code ownership...?

Re: Monorepo: please do

#50

Post by mindcrime »

klodolph wrote: post #16
benmarten wrote: post #3

Please don't. It's just too slow and not efficient. Instead use common open source best practices of shared library architecture. Problem solved! Putting everything into one repo is just lack of organization and creates a huge mess.

I feel like you've really done no work supporting your argument there. "Slow and inefficient"... what, exactly, is slow and inefficient? Because there are plenty of things slow and inefficient about polyrepos. I'd say that open-source best practices for shared libraries are appropriate if you're making an open-source shared library. However, these practices are inappropriate for internal libraries, proprietary librar…

However, these practices are inappropriate for internal libraries, proprietary libraries, and other use cases.

Why do you say so?

Post reply on HN