Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

121–130 of 268 posts

Re: The Ingredients of a Productive Monorepo

#121

Post by ellisv »

cousin_it wrote: post #104

I've worked for a company with a large monorepo. At first I was a fan, but now I'm not so sure. The web of dependencies was too much. Now I think teams should be allowed to reuse other teams' code only as libraries or APIs with actual release cycles. There shouldn't be any "oh let's depend on the HEAD of this random build target somewhere else in the monorepo". There should be only "let's depend on a released version…

I haven't really worked with any large monorepos.

I find your comment really interesting because having the capability to point to the HEAD (or realistically a commit SHA) is a feature I sometimes really enjoy about not using monorepos.

Re: The Ingredients of a Productive Monorepo

#123

Post by bluGill »

shawabawa3 wrote: post #19
echelon wrote: post #12

Earlier quoted context omitted.

As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

My company has millions of dollars in tooling for our polyrepo. It would not be hard to throw several more million into the problem.

If you have a large project there is no getting around the issues you will have. Just a set of pros and cons.

There are better tools for polyrepo you can start with, but there is a lot of things that we have that I wish I could get upstreamed (there is good reason the open source world would not accept our patches even if I cleaned them up)

Re: The Ingredients of a Productive Monorepo

#124

Post by lbhdc »

bittermandel wrote: post #73

I firmly believe that us at Molnett(serverless cloud) going for a strict monorepo built with Bazel has been paramount to us being able to make the platform with a small team of ~1.5 full-time engineers. We can start the entire platform, Kubernetes operators and all, locally on our laptops using Tilt + Bazel + Kind. This works on both Mac and Linux. This means we can validate essentially all functionality, even our Bo…

I am in a pretty similar situation as you, and have really been feeling the benefits of going all in on bazel.

> We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations.

Currently I have to run `bazel run `. Your solution sounds way better. How does yours work?

Re: The Ingredients of a Productive Monorepo

#125

Post by bluGill »

CamouflagedKiwi wrote: post #46
shawabawa3 wrote: post #19

Earlier quoted context omitted.

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

I think one reason is that there are various big companies (Google, Microsoft, Meta) who have talked about the tech they've deployed to make monorepos work, but I at least have never seen an equivalent big successful company describe their polyrepo setup, how they solved the pain points and what the tech around it looks like.

I keep meaning to write a blog post...

The short answer, start with a package management system like conan or npm (we rolled our own - releasing 1.0 the same month I first heard of conan which was then around version 0.6 - don't follow our example). Then you just need processes to ensure that everyone constantly has the latest version of all the repos they depend on - which ends up being a full time job for someone to manage.

Don't write your own package manager - if you use a common one that means your IDE will know how to work with it - our custom package manager has some nice features but we have to maintain our own IDE plugin so it can figure out the builds.

Re: The Ingredients of a Productive Monorepo

#126

Post by vegetablepotpie »

946789987649 wrote: post #81
gorgoiler wrote: post #65

An unspoken truth of a monorepo is that everyone is committed to developing on trunk, and trunk is never allowed to be broken. The consequence of this is that execution must be configurable at runtime: feature flags and configuration options with old and new code alongside each other. You can have a monorepo and still fail if every team works on their own branch and then attempts to integrate into trunk the week befo…

I didn't know places still had quarterly releases. That seems to like the one to resolve rather than a mono repo.

Every company that uses SAFe agile has quarterly, or bi-quarterly, releases [1].

[1] https://www.servicenow.com/docs/bundle/yokohama-it-business-...

Re: The Ingredients of a Productive Monorepo

#127

Post by eddd-ddde »

cousin_it wrote: post #104

I've worked for a company with a large monorepo. At first I was a fan, but now I'm not so sure. The web of dependencies was too much. Now I think teams should be allowed to reuse other teams' code only as libraries or APIs with actual release cycles. There shouldn't be any "oh let's depend on the HEAD of this random build target somewhere else in the monorepo". There should be only "let's depend on a released version…

This just creates tons of fragmentation. The second you have multiple teams depending on multiple versions you are doomed. You are stuck maintaining multiple versions, with their own quirks and bugs.

I think the one version rule is the most important part for a healthy monorepo.

Re: The Ingredients of a Productive Monorepo

#128

Post by grumpy_coder »

zelphirkalt wrote: post #74
nssnsjsjsjs wrote: post #53

Earlier quoted context omitted.

Monorepo monoliths make it hard to experiment. Getting something as benine as a later version of .NET becomes a mammoth task requiring the architecture team and everything stays old. Want to use a reasonable tool? No chance.

I don't see how it immediately has to follow from monorepo usage, that its parts cannot have separate runtimes and dependency versions. Perhaps the monorepo tooling is still that bad, idk, but there seems no inherent reason for that.

Monorepo very often means bazel for tooling (rbe and caching tests) and that means one WORKSPACE with common versions of libs.

Monorepo also means a team 'vetting' new thirdparty libs, and a team telling you your CI takes too long, and a team telling you to upgrade your lib within 23 minutes because theres a security issue in the korean language support...

Re: The Ingredients of a Productive Monorepo

#129

Post by bluGill »

cmrdporcupine wrote: post #103
bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're buildi…

That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith.

The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well.

The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really hard to get right (and Agile discourages it - which is right for small projects where those architects add complex things to mitigate problems you will never have)

Re: The Ingredients of a Productive Monorepo

#130

Post by eddd-ddde »

spankalee wrote: post #57

I love monorepos, but in large organizations they have a counter-intuitive incentive for teams to _not_ allow other teams to depend on them, which can _reduce_ code reuse - the opposite of what some adopters want. This issue is that users of a library can put almost infinite friction on the library. If the library team wants to make a change, they have to update all the use sites, but Hyrum's Law will get you because…

Well when making a library that's intended to be shared, you REALLY need to stop for a second and think about the API. Ideally APIs don't change, and when they do, you better have planned for large scale changes, or just use a new function and mark the old one deprecated.

I don't think there's anything wrong with copy pasting some useful piece of code too, not everything has to be a library you depend on, for small enough things.

Post reply on HN