Monorepo: please do • Hacker News Board

Re: Monorepo: please do

#151

Post by alexnewman » Fri, Jan 04, 2019, 5:03 PM UTC

benmarten wrote: post #4

sharpercoder wrote: post #2

In my experience, this discussion gets convoluted by confusing modularity with monorepo . They are orthogonal to each other; you can have a very modular codebase in a monorepo but also a very coupled (non-modular) codebase with polyrepo. Though it's true that monorepos without proper discipline can tend towards coupling. Yet, when discussing mono vs poly, we should keep this in mind.

It's not, why do I have to checkout terabyte of code that I don't need, even if the code is modularized?

Signs your build system is never going to be adopted outside of people cargo culting you?

  - [x] Namespaces and the like without much security benefit
  - [x] Giant Java dependency
  - [x] Strange syntax and glyphs

Re: Monorepo: please do

#152

Post by shados » Fri, Jan 04, 2019, 5:54 PM UTC

wikibob wrote: post #144

shados wrote: post #70

Earlier quoted context omitted.

> Why are you dependent on 3rd party code that isn't in your repo Not the parent, but for us, the 3rd party code is in a private package manager (artifactory, private npm, whatever). Having thousands of libraries we didn't write in our repo doesn't sound like fun.

Why are you using thousands of libraries?

The joke would be that even the simplest nodejs app these days pull in hundreds of libraries because of left-pad style libs...

The more serious answer is that when you have hundreds/thousands of applications with as many use cases, countless products and teams, and generally just ship a lot of stuff, it adds up.

Re: Monorepo: please do

#153

Post by Benjammer » Fri, Jan 04, 2019, 6:01 PM UTC

sebastos wrote: post #94

Benjammer wrote: post #31

Earlier quoted context omitted.

The real danger here is anyone talking about any system architectures or tooling as "dangerous" (or "not dangerous") absent any other context... What do you even mean by "dangerous"? To a business? To your health? What is the deal with people trying to make these sorts of global assertions in a vacuum about what's "good" and "bad"? This doesn't make any engineering sense in any way to me. You have a problem and you f…

Well just to present the other side, I don't really understand the prevalence of the "there's no one answer that fits for everybody" comment trope. You see a couple of comments like yours in every discussion like this. So no offense, but I'm going to rant about it for a few paragraphs. If the "no one-size-fits-all" claim happens to be genuinely and axiomatically true for a particular engineering trade-off, then fine.…

I'm of a mind that true understanding only really comes from questioning the most firmly held "universal truths." I want to know why I know what I know, I have no use for vague quality axioms that put black box abstractions on top of complex systems and processes.

> most people don't know most of the things you need to know to come up with a good answer to what suits "their specific situation".

And most people aren't competent software architects capable of adeptly steering an engineering team in the right choices to make. I'm not sure I understand the point here, or why you want to make a technical field like software engineering dumbed down to the point where "most people" can intuit the right decisions to make simply by asking HN what "the best thing" is.

Re: Monorepo: please do

#154

Post by senderista » Sat, Jan 05, 2019, 12:03 AM UTC

sigil wrote: post #131

holoway wrote: post #129

Earlier quoted context omitted.

Author here. The Right Way :tm: is situational - there isn't one right answer to things like when and how to make contracts, or how to break them. When I used the term "force", you'll see that I'm usually talking about dialog between people and teams. It's not that it's a single right way to do it. There isn't, and anyone who tells you there is has something to sell you, or is inexperienced enough to not have seen en…

Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.) Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs…

Actually Windows wasn’t a monorepo back then: there were separate repos for the shell, kernel, filesystem, etc. Hence the need for cross-repo tooling like “sdx”.

Source Depot was great (modulo availability issues), but I don’t think they got anywhere near the scale of Piper.

Re: Monorepo: please do

#155

Post by jacques_chester » Sat, Jan 05, 2019, 12:36 AM UTC

mlthoughts2018 wrote: post #142

jacques_chester wrote: post #99

Earlier quoted context omitted.

> simultaneous other transitions that are not logically required for any reason other than the superficial sake of the monorepo I don't see what's superficial about "everything everywhere is in sync", myself. And I have absolutely seen PR race conditions. Assuming that everyone perfectly sliced up the polyrepo on the first go is optimistic.

> “I don't see what's superficial about "everything everywhere is in sync", myself.” Well it is superficial by definition, because two unrelated things are “in sync” only because you say so. The very meaning of “in sync” in your sentence is some particular superficial standard you chose that has nothing to do with the logical requirements of the isolated subcomponents (i.e. “in sync” meaning two independent subcompon…

I work on a polyrepo. The code in repo A has a dependency on the code in repo B. When I update B, I sometimes need to update A.

In a monorepo that's already done when I finish working on the modules in B.

That I am unable to release from A until it has been synced with the module in B is not "a cosmetic notion". It's being unable to release. I consider releasability at all times to be the most important invariant to be sought by the combination of tests, CI and version control.

Re: Monorepo: please do

#156

Post by mlthoughts2018 » Sat, Jan 05, 2019, 12:44 AM UTC

jacques_chester wrote: post #155

mlthoughts2018 wrote: post #142

Earlier quoted context omitted.

> “I don't see what's superficial about "everything everywhere is in sync", myself.” Well it is superficial by definition, because two unrelated things are “in sync” only because you say so. The very meaning of “in sync” in your sentence is some particular superficial standard you chose that has nothing to do with the logical requirements of the isolated subcomponents (i.e. “in sync” meaning two independent subcompon…

I work on a polyrepo. The code in repo A has a dependency on the code in repo B. When I update B, I sometimes need to update A. In a monorepo that's already done when I finish working on the modules in B. That I am unable to release from A until it has been synced with the module in B is not "a cosmetic notion". It's being unable to release . I consider releasability at all times to be the most important invariant to…

> “In a monorepo that's already done when I finish working on the modules in B.”

This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use.

I worked in a monorepo for a long time where you still had to deploy versioned artifacts. So when you makes changes to B, you still have to bump version IDs, pass deployment requirements and upload the new version of B to internal pypi or internal maven or internal artifactory, etc.

Then consumer app A needs to update its version of B, test out that it works and that, from app A’s point of view, it is ready and satisfied to opt-in to B’s new changes, and do build + deploy of its own to deploy A with upgraded B.

Doing this in a way where a successful merge to master (or equivalent notion) of a change for B is suddenly a de facto upgrade for all the consumers of B is insanely bad for so many reasons that I’m not even going to try to list them all. Monorepo or not, nobody should be doing that, that is bonkers, crazy town bad. It’s a similar order of magnitude of bad as naively checking credentials into version control.

Re: Monorepo: please do

#157

Post by joshuamorton » Sat, Jan 05, 2019, 1:52 AM UTC

mlthoughts2018 wrote: post #156

jacques_chester wrote: post #155

Earlier quoted context omitted.

I work on a polyrepo. The code in repo A has a dependency on the code in repo B. When I update B, I sometimes need to update A. In a monorepo that's already done when I finish working on the modules in B. That I am unable to release from A until it has been synced with the module in B is not "a cosmetic notion". It's being unable to release . I consider releasability at all times to be the most important invariant to…

> “In a monorepo that's already done when I finish working on the modules in B.” This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use. I worked in a monorepo for a long time where you still had to deploy versioned artifacts. So when you makes changes to B, you still have to bump version IDs, pass de…

I think you're conflating wire format changes (which agreed, should be versioned and backwards compatible) with code level api changes. If V2 of xyz.h adds an argument to some method, a polyrepo just updates the tests and submits the change. In a monorepo, you can't submit until all clients are also updated.

Re: Monorepo: please do

#158

Post by mlthoughts2018 » Sat, Jan 05, 2019, 2:31 AM UTC

joshuamorton wrote: post #157

mlthoughts2018 wrote: post #156

Earlier quoted context omitted.

> “In a monorepo that's already done when I finish working on the modules in B.” This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use. I worked in a monorepo for a long time where you still had to deploy versioned artifacts. So when you makes changes to B, you still have to bump version IDs, pass de…

I think you're conflating wire format changes (which agreed, should be versioned and backwards compatible) with code level api changes. If V2 of xyz.h adds an argument to some method, a polyrepo just updates the tests and submits the change. In a monorepo, you can't submit until all clients are also updated.

No, even in a monorepo you can submit the code whenever you want, and have CI publish a versioned artifact from just that submodule / package / whatever. Other client code in the same repo can happily keep going along never caring about those new changes until later when explicitly ready to adopt them via adopting the new version.

There’s no reason why CI in a monorepo can’t create versioned code artifacts like Python packages, Java libraries or special jars, Docker containers, whatever. This is a very common workflow, e.g. combining a monorepo with in-house artifactory.

Definitely not talking wire format changes. Talking about publishing versioned libraries, jars, etc., from subsets of monorepo code.

Re: Monorepo: please do

#159

Post by luord » Sat, Jan 05, 2019, 2:58 AM UTC

Yet another chapter in one of the big flamewars. Seeing as I fall in the monorepo camp, I must say I mostly agree; also, I much prefer this tone for an article.

I find it enjoyable how plenty of comments both here and in the other discussion are of people saying "We had a mono/polyrepo and things improved tremendously when we migrated towards a poly/monorepo". The issue might be one of growth and complacency: a drastic change like that forces the team to face the technical debt that was being ignored and do a better implementation using what was learned from past mistakes.

Re: Monorepo: please do

#160

Post by senderista » Sat, Jan 05, 2019, 3:10 AM UTC

thurn wrote: post #40

totallysnowman wrote: post #19

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…

the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...

AMZN doesn’t, unless things have changed drastically in the last 3 years.