Monorepo: please do • Hacker News Board

Re: Monorepo: please do

#161

Post by sigil » Sat, Jan 05, 2019, 4:09 AM UTC

senderista wrote: post #154

sigil wrote: post #131

Earlier quoted context omitted.

Thanks for the response. Out of curiosity, how does your engineering organization introduce new dependencies within the monorepo? Can B, C and D all depend on A without A's consent or even awareness? (Suppose A is some checked in code that's useful, going to see updates in future, but is dormant at present.) Your post puts a lot of the onus on A for breaking B, C, and D, but I think equal care and consideration needs…

Actually Windows wasn’t a monorepo back then: there were separate repos for the shell, kernel, filesystem, etc. Hence the need for cross-repo tooling like “sdx”. Source Depot was great (modulo availability issues), but I don’t think they got anywhere near the scale of Piper.

Actually Windows wasn’t a monorepo back then: there were separate repos for the shell, kernel, filesystem, etc. Hence the need for cross-repo tooling like “sdx”.

This is a bit misleading to outsiders. Each of these repos was huge for the time, corresponded to a major subsystem with many disparate components, and the default tooling on the ground was the cross-repo tooling. One got the impression that if they could have pulled off one giant monorepo to rule them all, they would have, but they fell just short due to some technical details (cough spinning magnetic disks). In the meantime `sdx` was a convenient abstraction that allowed people to work in a monorepo way.

All in all it wasn't so different from present-day monorepos broken into git submodules for performance reasons.

Re: Monorepo: please do

#162

Post by laurentlb » Sat, Jan 05, 2019, 1:49 PM UTC

fxfan wrote: post #113

There's a lot of discussion of bazel and co inside sub-comments but i have a question that isn't addressed- How do the "global build tools" play with language specific build tools? My primary stack is Rust and Scala. Both have excellent build capabilities in their native tools. How well do pants/bazel integrate with them? I wouldn't want to rewrite complex builds nor would I expect these tools to have 100% functional…

Bazel has some level of support for many languages: https://docs.bazel.build/versions/master/be/overview.html#ad...

I know the Scala rules are used in production by multiple companies. Rust support is improving quickly, but it's not perfect. See the dedicated GitHub repositories for more information.

(I work on Bazel)

Re: Monorepo: please do

#163

Post by jacques_chester » Sat, Jan 05, 2019, 3:37 PM UTC

mlthoughts2018 wrote: post #156

jacques_chester wrote: post #155

Earlier quoted context omitted.

I work on a polyrepo. The code in repo A has a dependency on the code in repo B. When I update B, I sometimes need to update A. In a monorepo that's already done when I finish working on the modules in B. That I am unable to release from A until it has been synced with the module in B is not "a cosmetic notion". It's being unable to release . I consider releasability at all times to be the most important invariant to…

> “In a monorepo that's already done when I finish working on the modules in B.” This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use. I worked in a monorepo for a long time where you still had to deploy versioned artifacts. So when you makes changes to B, you still have to bump version IDs, pass de…

> This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use.

I guess compilers must work differently for you.

Re: Monorepo: please do

#164

Post by mlthoughts2018 » Sat, Jan 05, 2019, 4:31 PM UTC

jacques_chester wrote: post #163

mlthoughts2018 wrote: post #156

Earlier quoted context omitted.

> “In a monorepo that's already done when I finish working on the modules in B.” This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use. I worked in a monorepo for a long time where you still had to deploy versioned artifacts. So when you makes changes to B, you still have to bump version IDs, pass de…

> This is not usually true in monorepos or polyrepos, and is quite a dangerous practice that nobody should use and hasn’t got much at all to do with what type of repo you use. I guess compilers must work differently for you.

Huh? This comment makes no sense. You just use a build file definition to pull in the version of the compiled artifacts you need according to whichever version you want, no different than how you would include third party dependencies.

For example, in the most recent monorepo I worked in, most everything was written in Java and Scala. But when you compile consumer app A that depends on submodule B, it does not just naively use the code of submodule B already sitting at the same commit of the monorepo. That would be terrible, because it would mean if anyone changed some code in submodule B, then app A has been silently upgraded just by default.

Instead, the necessary shared object / jar / whatever is compiled only for submodule B, which is then uploaded with its new version identifer to internal artifactory that stores the compiled jars, shared objects, whatever (and stores Python packages, containers, and many other types of artifacts too).

Now when you compile app A, it retrieves the right artifacts it needs from artifactory, to treat submodule B like a totally separate third party library, and app A is free to specify whatever version of B that it needs, no different than specifying open source third party dependencies.

It really seems like you are willfully trying to act like you don’t understand what I’m saying. This approach works perfectly for compiled languages and artifacts, that’s one of the primary use cases it is designed from the ground up to solve.

Re: Monorepo: please do

#165

Post by totallysnowman » Sat, Jan 05, 2019, 5:14 PM UTC

woolvalley wrote: post #35

totallysnowman wrote: post #19

I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…

Why does the build not work at 500 engineers? Just because there are 5 apps / services and their 25 libraries doesn't mean you have build the 4 other apps when doing development just because you did a git pull. Libraries would still need to build their dependants in a CI system with poly or mono repos.

The 5 apps and their 25 libraries are in fact one entangled monolith. That is what I am talking about. It is only marketed as 5 apps. That has some hundreds external dependencies, of course.

Re: Monorepo: please do

#166

Post by joshuamorton » Sat, Jan 05, 2019, 7:53 PM UTC

mlthoughts2018 wrote: post #158

joshuamorton wrote: post #157

Earlier quoted context omitted.

I think you're conflating wire format changes (which agreed, should be versioned and backwards compatible) with code level api changes. If V2 of xyz.h adds an argument to some method, a polyrepo just updates the tests and submits the change. In a monorepo, you can't submit until all clients are also updated.

No, even in a monorepo you can submit the code whenever you want, and have CI publish a versioned artifact from just that submodule / package / whatever. Other client code in the same repo can happily keep going along never caring about those new changes until later when explicitly ready to adopt them via adopting the new version. There’s no reason why CI in a monorepo can’t create versioned code artifacts like Pytho…

Why would you do versioning in a monorepo?

>There’s no reason why CI in a monorepo can’t create versioned code artifacts like Python packages, Java libraries or special jars, Docker containers, whatever. This is a very common workflow, e.g. combining a monorepo with in-house artifactory.

Correct, and this is necessary. But there's no reason for a to depend on b from the artifactory instead of a just depending on b at a source level, and building a and b simultaneously and linking them together. Now you have fully hermetic, reproducible builds and tests.

Why is not doing versioning so insanely bad that you can't list all the reasons (this would be a much more interesting discussion if you did).