Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

141–150 of 268 posts

Re: The Ingredients of a Productive Monorepo

#141

Post by surajrmal »

946789987649 wrote: post #81
gorgoiler wrote: post #65

An unspoken truth of a monorepo is that everyone is committed to developing on trunk, and trunk is never allowed to be broken. The consequence of this is that execution must be configurable at runtime: feature flags and configuration options with old and new code alongside each other. You can have a monorepo and still fail if every team works on their own branch and then attempts to integrate into trunk the week befo…

I didn't know places still had quarterly releases. That seems to like the one to resolve rather than a mono repo.

Android is only recently switching to quarter releases instead of yearly. Most. Popular Linux distros only have major releases every 6 months. While chrome cuts a release branch every 4 weeks, it soaks it in a beta channel for another 4. Same goes for the rust compiler toolchain, albeit on a 6 week cadence.

Re: The Ingredients of a Productive Monorepo

#142

Post by dalyons »

zbentley wrote: post #116
CamouflagedKiwi wrote: post #44

Earlier quoted context omitted.

I do think the 'run all tests on CI' part is not that fine, it bites a lot earlier than the others do. Git is totally fine for a few hundred engineers and 100ish services (assuming nobody does anything really bad to it, but then it fails for 10 engineers anyway), but running all tests rapidly becomes an issue even with tens of engineers. That is mitigated a lot by a really good caching system (and even more by full r…

I think that discussions in this area get muddied by people using different definitions of “rapidly”. There are (at least) two kinds of speed WRT tests being run for a large code base. First, there is “rapidly” as pertains to the speed of running tests during development of a change . This is “did I screw up in an obvious way” error checking, and also often “are the tests that I wrote as part of this change passing”…

You are of course entitled to your opinion, and I do appreciate going against the grain, but having worked in an “hours” environment and a “minutes” environment I couldn’t disagree more. The minutes job is so much more pleasant to work with in nearly every way. And ironically ended up being higher quality because you couldn’t lean on a giant integration test suite as a crutch. Automated business metric based canary rollbacks, sophisticated feature flagging and gating systems, contract tests, etc. and these run in production, so are accurate where integration tests often aren’t in a complicated service topology.

There are also categories of work that are so miserable with long deployment times that they just don’t get done at all in those environments. Things like improving telemetry, tracing, observability. Things like performance debugging, where lower envs aren’t representative.

I would personally never go back, for a system of moderate or more distributive complexity (ie > 10 services, 10 total data stores )

Re: The Ingredients of a Productive Monorepo

#143

Post by dezgeg »

zelphirkalt wrote: post #72
dezgeg wrote: post #39

Earlier quoted context omitted.

> Take your monorepo, split it into n repos by directory at certain depth from root, write very a rudimentary VCS wrapper script to sync all the repos in tandem and you have already solved a lot of pain points. Then you lose the capability to atomically make a commit that crosses repoes. I'm not sure if there is any forge that allows that, except Gerrit might with its topics feature (I've not gotten the opportunity t…

You could also use git submodules in an overarching separate repo, if you want to lock down a set of versions. It doesn't even have to affect the submodule repos in any way. That would simplify branches in the single repos and enable teams to work independently on each repo. Then you only deploy from the overarching repo's main branch for example, where you have to create PRs for merging into the main branch and get…

That's not a nice workflow from pipelines/CI point of view.

Let's take for example a service 'foobar' that depends on in-house library 'libfoo'. And now you need to add a feature to foobar that needs some changes to libfoo at same time (and for extra fun let's say those changes will break some other users of libfoo). Of course during development you want to run pipelines for both libfoo and foobar.

In such 'super module' system it gets pretty annoying to push changes for testing in CI when every change to either libfoo or foobar needs to be followed by a commit to the super repo.

In a monorepo that's just another Tuesday.

Re: The Ingredients of a Productive Monorepo

#144

Post by lolinder »

grumpy_coder wrote: post #128
zelphirkalt wrote: post #74

Earlier quoted context omitted.

I don't see how it immediately has to follow from monorepo usage, that its parts cannot have separate runtimes and dependency versions. Perhaps the monorepo tooling is still that bad, idk, but there seems no inherent reason for that.

Monorepo very often means bazel for tooling (rbe and caching tests) and that means one WORKSPACE with common versions of libs. Monorepo also means a team 'vetting' new thirdparty libs, and a team telling you your CI takes too long, and a team telling you to upgrade your lib within 23 minutes because theres a security issue in the korean language support...

Monorepo doesn't mean any of those things, nor does a polyrepo setup prevent any of them except for bazel.

It sounds like you worked in a dysfunctional organization that happened to use a monorepo. Their dysfunctions are not inherent in the monorepo model and they would have found other ways to be dysfunctional if not those.

Re: The Ingredients of a Productive Monorepo

#145

Post by halpow »

cousin_it wrote: post #104

I've worked for a company with a large monorepo. At first I was a fan, but now I'm not so sure. The web of dependencies was too much. Now I think teams should be allowed to reuse other teams' code only as libraries or APIs with actual release cycles. There shouldn't be any "oh let's depend on the HEAD of this random build target somewhere else in the monorepo". There should be only "let's depend on a released version…

[dead]

Re: The Ingredients of a Productive Monorepo

#146

Post by mherkender »

cmrdporcupine wrote: post #103
bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're buildi…

Large software projects cycle back and forth between fragmentation and defragmentation. There is no right answer, only what's right for each project at the time.

Relevant xkcd: https://xkcd.com/2044/

Re: The Ingredients of a Productive Monorepo

#147

Post by hiddencost »

przmk wrote: post #82
lxe wrote: post #8

So there are 2 kinds of big tech monorepos. One is the kind described in the article here: "THE" monorepo of the (mostly) entire codebase, requiring custom VCS, custom CI, and a team of 200 engineering supporting this whole thing. Uber and Meta and I guess Google do it this way now. It takes years of pain to reach to this point. It usually starts with the other kind of "monorepo": The other kind is the "multirepo mon…

At my current $dayjob, there is a backend that is split into ~11 git repos which results in a single feature being split among 4-5 merge requests and it's very annoying. We're about to begin evaluating monorepos to group them all (among other projects). What would the alternative to a monorepo be in this case, knowing that we can't bundle the repos together?

Do these 11 repos end up in separate binaries?

Because it sounds like you just need flag based feature releases.

Re: The Ingredients of a Productive Monorepo

#148

Post by friendzis »

dezgeg wrote: post #143
zelphirkalt wrote: post #72

Earlier quoted context omitted.

You could also use git submodules in an overarching separate repo, if you want to lock down a set of versions. It doesn't even have to affect the submodule repos in any way. That would simplify branches in the single repos and enable teams to work independently on each repo. Then you only deploy from the overarching repo's main branch for example, where you have to create PRs for merging into the main branch and get…

That's not a nice workflow from pipelines/CI point of view. Let's take for example a service 'foobar' that depends on in-house library 'libfoo'. And now you need to add a feature to foobar that needs some changes to libfoo at same time (and for extra fun let's say those changes will break some other users of libfoo). Of course during development you want to run pipelines for both libfoo and foobar. In such 'super mod…

> In such 'super module' system it gets pretty annoying to push changes for testing in CI when every change to either libfoo or foobar needs to be followed by a commit to the super repo.

Again, tooling issue. CI can easily pull required changeset across multiple repos. We are in a subthread under "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo"

Re: The Ingredients of a Productive Monorepo

#149

Post by przmk »

dustingetz wrote: post #92
przmk wrote: post #82

Earlier quoted context omitted.

At my current $dayjob, there is a backend that is split into ~11 git repos which results in a single feature being split among 4-5 merge requests and it's very annoying. We're about to begin evaluating monorepos to group them all (among other projects). What would the alternative to a monorepo be in this case, knowing that we can't bundle the repos together?

use git subtree - first to concatenate the minor repos into one major repo, and then subtree split from that point forward to publish subtree folders back to the minor repos, if needed (e.g. open source projects to github). works for us with about 8 minor repos, eliminated submodule pain entirely. only the delivery lead has to even know the minor repos exist.

I have already briefly looked at git-subtree. From what I can gather, it doesn't help much with my use-case. You still need to manually pull from each subtree and push branches individually to each project. The end result is still 4-5 merge requests to handle on Gitlab for a single new feature.

I might have missed something.

Re: The Ingredients of a Productive Monorepo

#150

Post by przmk »

hiddencost wrote: post #147
przmk wrote: post #82

Earlier quoted context omitted.

At my current $dayjob, there is a backend that is split into ~11 git repos which results in a single feature being split among 4-5 merge requests and it's very annoying. We're about to begin evaluating monorepos to group them all (among other projects). What would the alternative to a monorepo be in this case, knowing that we can't bundle the repos together?

Do these 11 repos end up in separate binaries? Because it sounds like you just need flag based feature releases.

It ends up with 6 deployables that are coupled together (let's say micro-services). There are surely better ways to structure the project but our CI/CD pipeline doesn't allow us to do so and it is not handled by our team anyway. I haven't seen any good way to make my life easier for merges, tech reviews, deployments, etc…
Post reply on HN