Monorepo: please do • Hacker News Board

Re: Monorepo: please do

#101

Post by albedoa » Fri, Jan 04, 2019, 2:13 AM UTC

woolvalley wrote: post #32

My org went from polyrepo 10 commit semver dependency hell when updating an internal API to monorepo and it saves a lot of time. Unmigrated semver breaking changes are a form of technical debt, and it takes a lot more total man hours to do the 'proper' one by one many commit poly repo migration than the other way around. If we had the tooling to do multirepo atomic commits and reviews then maybe we would of stuck wit…

> Unmigrated semver breaking changes are a form of technical debt

Maybe you can clear my confusion. If Module B is dependent on Module A, then every version of B should refer to a specific version of A, correct? What is there to break? Development can continue on A without interfering with B, and then you can uptick B once it points to a later A.

I'm not sure what this has to do with the mono/poly discussion.

Re: Monorepo: please do

#102

Post by jacques_chester » Fri, Jan 04, 2019, 2:13 AM UTC

jchw wrote: post #97

jacques_chester wrote: post #72

Earlier quoted context omitted.

I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with e…

It is not a smooth experience outside of Google because the truth is bootstrapping a proper Bazel setup is not actually that easy. If you want hermetic builds for real, you need a hermetic build environment. Bazel tries to accomplish this with a workspace setup in each repo, but unfortunately it's definitely limited and imperfect. The Bazel rules for languages is also not perfect imo. Like I dislike hooking Bazel up…

> However, here's what I will say: if you are in an organization, I think Bazel really shines. If you can take time to write some custom tools and rules and really integrate your software into Bazel, it can be an awesome experience. ... Another plus: it is Amazing having a build system that crosses languages.

This is pretty much what I think of when I want to like Bazel. I wish we had it on Cloud Foundry. Or, rather, I wish it had existed 5 years ago and had been used on Cloud Foundry from the beginning, because CF and its associated projects have hundreds of repositories and these have mostly been kept in sync through mountains of tests and oceans of automation. It works, but I know that in another universe it works better.

Re: Monorepo: please do

#103

Post by ceronman » Fri, Jan 04, 2019, 2:15 AM UTC

skybrian wrote: post #24

I wonder if a star pattern would work, where you have a single, shared repo for all your libraries and a repo for each app. This would help people working on smaller apps, since they don't need to look at other apps unless they're working on shared library code. Of course, once you are working on library code, you have to build and test all the apps that use it. But even at Google, the people working on the lowest le…

A star pattern still has most of the downsides of the multirepo approach. Specifically, it has the problem of needing a parallel version control (e.g. SemVer) on top of your individual repositories. This creates fragmentation, where different applications have dependencies on different versions of the libraries which ends up in dependency hell, technical debt, and CI hell.

Re: Monorepo: please do

#104

Post by sixstringtheory » Fri, Jan 04, 2019, 2:15 AM UTC

jacques_chester wrote: post #75

mindcrime wrote: post #44

"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it do…

> As the old saying goes "when everybody is responsible, nobody is responsible". Here's my rule: You break it, you fix it. > I'd rather see a poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR. I'd rather see pairing, extensive tests and fast CI. I see PRs as a necessary evil, rather than a good thing in themselves. If I make a…

> You break it, you fix it.

I’ve seen this invoked so many times to shirk responsibility though. Someone piles up all kinds of crap in a tight little closet, complete with a bowling ball on top, and the next unsuspecting dev who comes by and opens it gets an avalanche of crap falling on them while the original author can be heard somewhere in the background saying “it’s not my problem.”

This winds up leading to more crap-stacking just to get the work done ASAP and you wind up with a mountain of tech debt.

I like the zero flaw principle where new feature work stops until all currently known flaws are fixed. Then everyone is forced to pitch in and responsibility is shared whether you want it or not.

Re: Monorepo: please do

#105

Post by jacques_chester » Fri, Jan 04, 2019, 2:15 AM UTC

jkaplowitz wrote: post #86

jacques_chester wrote: post #82

Earlier quoted context omitted.

bazel build //main:hello-world I'm sure the double slashes and colon have important differences. It is not obvious what they are. cc_binary( name = "hello-world", srcs = ["hello-world.cc"], deps = [ ":hello-greet", "//lib:hello-time", ], ) It's not instantly obvious why one is :hello-greet and the other is //lib:hello-time. I could swear I've seen @ floating around as well. As I said above, I am sure these are all ve…

Completely valid concern not to want to keep memorizing mini-languages. In this case, the double slashes are absolute "paths" relative to the top of the workspace, and the part after the colon is a relative "path" to another Bazel target. I put "paths" in quotes because these are meaningfully different from the true filesystem equivalents; avoiding confusion with real absolute and relative filesystem paths is probabl…

> What would the better way have been for them to do this?

I don't know, off the top of my head (having been on the other side of this conversation, I am aware how frustrating that answer is). But I know I couldn't keep it straight when I was fighting Bazel and that I gave up. And anecdotally I am not alone: I have seen Bazel torn out of multiple projects, sometimes quite painfully.

Re: Monorepo: please do

#106

Post by jacques_chester » Fri, Jan 04, 2019, 2:17 AM UTC

sixstringtheory wrote: post #104

jacques_chester wrote: post #75

Earlier quoted context omitted.

> As the old saying goes "when everybody is responsible, nobody is responsible". Here's my rule: You break it, you fix it. > I'd rather see a poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR. I'd rather see pairing, extensive tests and fast CI. I see PRs as a necessary evil, rather than a good thing in themselves. If I make a…

> You break it, you fix it. I’ve seen this invoked so many times to shirk responsibility though. Someone piles up all kinds of crap in a tight little closet, complete with a bowling ball on top, and the next unsuspecting dev who comes by and opens it gets an avalanche of crap falling on them while the original author can be heard somewhere in the background saying “it’s not my problem.” This winds up leading to more…

> I’ve seen this invoked so many times to shirk responsibility though. Someone piles up all kinds of crap in a tight little closet, complete with a bowling ball on top, and the next unsuspecting dev who comes by and opens it gets an avalanche of crap falling on them while the original author can be heard somewhere in the background saying “it’s not my problem.”

I'm accustomed to collective ownership where, ideally, this never happens and in practice happens rarely (followed by the little closet being torn out and replaced).

> I like the zero flaw principle where new feature work stops until all currently known flaws are fixed.

I agree: stop the line. But I think it's orthogonal to the sins or virtues of n-repology.

Re: Monorepo: please do

#107

Post by Tempest1981 » Fri, Jan 04, 2019, 2:19 AM UTC

For those using monorepos, what is your branch strategy? Say that 3 projects share a library, and release on different schedules. How does each project freeze shared library changes? Do you keep N version branches?

How does the library team know which consumers a commit may break? What tools are recommended?

Re: Monorepo: please do

#108

Post by [deleted] » Fri, Jan 04, 2019, 2:30 AM UTC

sebastos wrote: post #94

Benjammer wrote: post #31

Earlier quoted context omitted.

The real danger here is anyone talking about any system architectures or tooling as "dangerous" (or "not dangerous") absent any other context... What do you even mean by "dangerous"? To a business? To your health? What is the deal with people trying to make these sorts of global assertions in a vacuum about what's "good" and "bad"? This doesn't make any engineering sense in any way to me. You have a problem and you f…

Well just to present the other side, I don't really understand the prevalence of the "there's no one answer that fits for everybody" comment trope. You see a couple of comments like yours in every discussion like this. So no offense, but I'm going to rant about it for a few paragraphs. If the "no one-size-fits-all" claim happens to be genuinely and axiomatically true for a particular engineering trade-off, then fine.…

[deleted]

Re: Monorepo: please do

#109

Post by ceronman » Fri, Jan 04, 2019, 2:32 AM UTC

I work at a large organization (2000+ devs). We have used both a Monorepo and Polyrepo. After some extensive experience with both models my conclusion is that the Monorepo is by far a superior model, specially for a large organization.

Of course the Monorepo is not free of downsides, those mentioned in the original article are real, although a bit exaggerated in my opinion. VCS operations can be slow and scaling a VCS system is challenging, but possible. And the risk of high coupling and a tangled architecture is also very real if you don't use a dependency management system like Bazel/Buck/Pants.

But in my opinion the downsides of the Polyrepo are much worse and much much harder to fix. The main problem is that you need a parallel version control system like SemVer on top of your VCS. SemVer is fine for open source projects but for a dynamic organization is a nightmare because it is a manual process prone to failure. SemVer dependency hell is really hard to deal with and creates a lot of technical debt.

Additionally, once you go Polyrepo you lose true CI/CD. Yes, you still have CI/CD pipelines but those apply only to a fraction of the code. Once you get used to run `bazel test` and you know you will run every single test of any piece of code that could depend on the code you just changed, you never want to go back. Yes, you could have true CI/CD with Polyrepos, but it requires a lot of work and writing a lot of tooling that does not exist in the wild. It is cheaper to invest in scaling your VCS in a multi-repo.

Re: Monorepo: please do

#110

Post by jacques_chester » Fri, Jan 04, 2019, 2:36 AM UTC

klodolph wrote: post #61

mindcrime wrote: post #50

Earlier quoted context omitted.

However, these practices are inappropriate for internal libraries, proprietary libraries, and other use cases. Why do you say so?

Basically because for certain projects and teams, the effort to package internal / proprietary libraries and other similar dependencies can be much larger than the benefit. Packaging is effort. You decide to cut a release, stamp a version number, write a changelog, package and distribute it, and then backport fixes into a long-running branch. This effort makes a lot of sense if your consumers are complete strangers w…

I broadly agree with all of this, though I think it's possible to simplify the business of packaging and releasing with the right automation. But lowering the cost doesn't change the more important question of whether that cost is worth bearing.

> One of the main benefits of version numbers is that it tells your consumers where the breaking changes are, but if you have direct access to your consumers’ code and can commit changes, review them, and run their CI tests, then you have something much better than version numbers.

A small peeve of mine: Semver and version numbers generally are lossy compression. They try to squeeze a wide range of information into a very narrow space, for no other reason than tradition.