Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

221–230 of 268 posts

Re: The Ingredients of a Productive Monorepo

#221

Post by crazygringo »

yencabulator wrote: post #211
crazygringo wrote: post #209

Earlier quoted context omitted.

That's confusing two different things, though. A monorepo does make changes atomic in the code. There's no trap there. You're talking about deployment, and yes when deployment is staggered, then obviously all atomic changes need to be backward-compatible, or else be very carefully orchestrated. But that doesn't have anything to do with monorepo vs polyrepo. That's just staggered deployment. You have to deal with back…

There's no such thing as when deployment is staggered. It's a distributed problem, so by definition it is not synchronous. (Or you turn services off for the duration of the deploy. Most companies do not want that these days.) Also, you're missing this part of the article: > While this is also possible in a world with many repositories, the requirement to do this change in multiple pull requests is often enough to rem…

I was talking about the concept of deployment in general. Yes, of course it's usually staggered for monorepos. I don't know why you're arguing about saying there's "no such thing as when" and then immediately point out a case of "when" immediately following. (And turning off services for 15 minutes at 2 am is definitely still a thing.)

And I'm not missing any part of the article. I was talking about your comment, and the fact that you are conflating two different things. A monorepo allows you to make atomic commits that a polyrepo does not, full stop. There's no trap there. Deployment is separate. You have to worry about breaking changes regardless of it being a polyrepo or monorepo. But a monorepo can make each atomic change far easier to track and manage and reason about.

Re: The Ingredients of a Productive Monorepo

#222

Post by codethief »

gengstrand wrote: post #184
bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

It is true that there are significant benefits to monorepo but it comes at a cost. Managing a monorepo is more expensive than polyrepo. For the details behind that claim, check out https://www.exploravention.com/blogs/monorepo/ The question is this. Do the costs of monorepo justify the benefits for your situation? The answer is not always yes.

Note that the costs depend on the scale and size of the monorepo, and a polyrepo is not without costs either (which people often like to forget or ignore because they are less visible).

I spent some years at small to mid-sized companies (~30-100 devs) that would have profited from a monorepo. However, people were in the habit of splitting repositories every other month. Sometimes a single team would own a given repository, but more often than not several teams would contribute to each repository.

I have serious PTSD from that time. Every single pipeline in every repo worked differently, tools were different, setup, scripts and commands were different. In some repositories you could trust the CI pipeline, in others you absolutely couldn't. CI performance gains in one repo wouldn't translate to another. And of course you often still had some tech debt lying around from when repositories got split and people forgot to clean up thoroughly. Uggh.

Now, to be fair, people did put quite a bit of effort into their setups and pipelines and all that, and it wasn't that the outcome of that effort in and by itself was totally bad. Not at all. But everyone did things differently. And often teams just re-did the work other people had already done & solved – there was a ton of overhead.

Finally, the worst part were inter-repository dependencies. People argued coupling would be loose and all that, so we could easily split repos, but in reality there were so many implicit dependencies because ultimately all repositories made up one big application. Merge requests had to be coordinated. One repo imported files from another. Later a CI pipeline in one repo triggered a pipeline in another repo…

This brings me to another problem with polyrepos: They harm discoverability and cause certain dependencies not to be encoded in code. In a monorepo, in contrast, when people wonder where they can find X, or whether anyone uses or depends on Y, the answer is usually only a `ripgrep` or `git grep` away. In a polyrepo, however, they need to know where to look first.

Born from all these experiences, my mantra has been: Split repos only if you really have to and have a very good reason! (Entirely different applications or ecosystems; different parts of the code owned by different & independent teams; …)

Re: The Ingredients of a Productive Monorepo

#223

Post by codethief »

shawabawa3 wrote: post #19
echelon wrote: post #12

Earlier quoted context omitted.

As a former IC at a large monorepo company, I preferred monorepos over polyrepos. It was the "THE" monorepo, and it made understanding the company's service graph, call graph, ownership graph, etc etc. incredibly clear. Crystal clear. Vividly so. Polyrepos are tribal knowledge. You don't know where anything lives and you can't look or discover it. Every team does their own thing. Inheriting new code is a curse. Code…

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

> Why can't we add millions of dollars of tool engineering on top of polyrepos

I don't think the "stock polyrepo" characterization is apt. Organizations using polyrepos already do invest that kind of money. Unfortunately, this effort is not visible because it's spread out across repos and every team does their own thing. So then people erroneously conclude that monorepos are much more expensive. Like the GP said:

> Polyrepos are a thousand species living and dying, some thriving, some never to be known, most completely in the dark.

Re: The Ingredients of a Productive Monorepo

#224

Post by codethief »

swgillespie wrote: post #6
Flux159 wrote: post #3

This definitely tracks with my experience in big tech - managing large scale build systems ends up taking a team that works on the build system itself. The underlying repo technology itself needs to work at scale & that was with a virtual file system that downloaded source files on demand when you needed to access them. One thing that this article didn't mention is that most development was done either on your develo…

Yeah - I worked on that build team probably at the same place you did! I think for a lot of users it's more important that the monorepo devenv be reproducible than be specifically local or specifically remote. It's certainly easier to pull this off when it's a remote devserver that gets regularly imaged.

> Yeah - I worked on that build team probably at the same place you did!

I did not work at that place but the story sounds very familiar – I believe there might have been a blog post about that remote development environment here on HN some time ago?

Re: The Ingredients of a Productive Monorepo

#225

Post by codethief »

peterldowns wrote: post #136
lbhdc wrote: post #124

Earlier quoted context omitted.

I am in a pretty similar situation as you, and have really been feeling the benefits of going all in on bazel. > We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations. Currently I have to run `bazel run `. Your solution sounds wa…

Not the OP but you can use tools like direnv + mise/asdf/nix so that every time a developer cd's into the monorepo, their shell environment loads a pinned, declaratively-configured set of dependencies and tools whose definitions are part of the monorepo. The way I'd naively set up something like OP described would be to have direnv + nix flake deliver you a copy of bazelisk, and then have some custom shell scripts ad…

Came here to post this. I've used asdf/mise for years (and recently also Nix + direnv) and it works tremendously well.

Re: The Ingredients of a Productive Monorepo

#227

Post by codethief »

cloogshicer wrote: post #47

Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?

There's a second option, not mentioned by the sibling comments so far: Publish the library somewhere (e.g. an internal Nexus) and then have the other consumer pin the old version of the library instead of referring to the newest version inside the monorepo. Whether or not this is acceptable is largely a question of ownership.

Re: The Ingredients of a Productive Monorepo

#228

Post by lbhdc »

mikn wrote: post #140
lbhdc wrote: post #124

Earlier quoted context omitted.

I am in a pretty similar situation as you, and have really been feeling the benefits of going all in on bazel. > We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations. Currently I have to run `bazel run `. Your solution sounds wa…

Hi! Previously mentioned ex-Google SRE! There are a few layers to it - to make it work "ok" you need to first have a tool runner wrapper rule that does something similar to: ``` ctx.actions.write(output=""" tool_path=$(realpath {tool_short_path}) cd ${{BUILD_WORKING_DIRECTORY}} exec $tool_path """.format(tool_short_path=tool.short_path) ``` The purpose of this rule is to ensure that the tool's CWD is actually where y…

Thanks! I am gonna give this a try.

Re: The Ingredients of a Productive Monorepo

#229

Post by sampullman »

CamouflagedKiwi wrote: post #151
sampullman wrote: post #118

Earlier quoted context omitted.

It's a bit of a tangent and I agree with your point, but wanted to note that for one project our e2e tests went from ~40 min to less than 10, just by moving from Cypress to Playwright. You can go pretty far with Playwright and a couple of cheap runners.

I appreciate the point, but I've heard this kind of thing several times before - last time around was hype about how Cypress would have exactly this effect (spoiler: it did not live up to the hype). I don't believe the new framework du jour will save you from this kind of thing, it's about how you write & maintain the tests.

I wish I had hard evidence to show because my normal instinct would be similar to yours, but in this case I'm a total Playwright convert.

Part of it might be that Playwright makes it much easier to write and organize complex tests. But for that specific project, it was as close to a 1 to 1 conversion as you get, the speedup came without significant architectural changes.

The original reason for switching was flaky tests in CI that were taking way too much effort to fix over time, likely due to oddities in Cypress' command queue. After the switch, and in new projects using Playwright, I haven't had to deal with any intermittent flakiness.

Re: The Ingredients of a Productive Monorepo

#230

Post by nobodywillobsrv »

bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

While things like git submodules have problems, are they not sort of the best of both worlds camp?

I never quite got the arguments/grumblings about this stuff but perhaps I do not understand what friction people are hitting up agains. It feels to me the problems of multiple repositories is the same problem of having to manage versions of dependencies like in pip or something that you do not even own.

Perhaps people are using tools that mean they are not able to search/click through their multi repo setup even though the code connects?

Post reply on HN