Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

171–180 of 268 posts

Re: The Ingredients of a Productive Monorepo

#171

Post by ecoffey »

cmrdporcupine wrote: post #103
bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're buildi…

In my experience microservices are easier to manage and understand when organized in a monorepo.

Re: The Ingredients of a Productive Monorepo

#172

Post by tex0 »

jbverschoor wrote: post #36
Pawka wrote: post #27

Earlier quoted context omitted.

It depends on the VCS you use. I don't know any ways to manage read permissions, such as allowing a person to checkout one directory but not another, though you can do that per branch on git. But there are many ways to manage write permissions - limit the directories to which engineers are allowed to push code. E.g. if you use Git, this can be done with Gitolite, which is a popular hosting server. Gitolite has very f…

It's mostly about read/access permissions. I'd like to stay away from any type of git hook tbh

Gerrit can do some of that.

Re: The Ingredients of a Productive Monorepo

#173

Post by bluGill »

dezgeg wrote: post #161
bluGill wrote: post #139

Earlier quoted context omitted.

If a single language is an option you are a small project that is not facing the problems people on large projects are facing. A monorepo will be easy for you without read the article and the lessons learned. Come back when you have millions of lines of code, written over decades by hundreds (or thousands) of full time developers.

What a weird take, "millions of lines of code, written over decades" applies to quite many C (or C++) codebases where using a high-level language is not a possibility (and companies that do have such codebases are pretty conservative and don't even talk about Rust no matter how great fit it would be).

In every case I've seen the vast majority might be C, but there are other other languages hidden in there that are hard to find. Many companies would use more languages if it wasn't such a pain. Rust for example would be really nice to use in new code, if only they can figure out how to mix it in.

Re: The Ingredients of a Productive Monorepo

#174

Post by bluGill »

kccqzy wrote: post #165
bluGill wrote: post #152

Earlier quoted context omitted.

That is a downside of a polyreop that you will need to figure out how to mitigate. It doesn't matter if you go monorepo or polyrepo you wil have issues as your project grows. You will need to mitigate those issues somehow.

In a polyrepo it is common to say I depend on this specific git SHA of that other repo. In a monorepo it is weird and unheard of to say I depend on this specific SHA of the current repo. It's a matter of defaults.

In a polyrepo you need to figure out how/when to update those SHAs. This is one of the hard things about polyrepos. Monorepo of course doesn't need that concept because you cannot depend on some previous state of the repo.

Re: The Ingredients of a Productive Monorepo

#176

Post by mamcx »

lxe wrote: post #8

So there are 2 kinds of big tech monorepos. One is the kind described in the article here: "THE" monorepo of the (mostly) entire codebase, requiring custom VCS, custom CI, and a team of 200 engineering supporting this whole thing. Uber and Meta and I guess Google do it this way now. It takes years of pain to reach to this point. It usually starts with the other kind of "monorepo": The other kind is the "multirepo mon…

And for small teams, what we want/need is the "all deps" mono-repo.

I wanna link other repos I depend on, but that repos can be read-only. And then all the tools work without extra friction

P.D: This could be another wish for jj!

Re: The Ingredients of a Productive Monorepo

#177

Post by no_wizard »

lxe wrote: post #8

So there are 2 kinds of big tech monorepos. One is the kind described in the article here: "THE" monorepo of the (mostly) entire codebase, requiring custom VCS, custom CI, and a team of 200 engineering supporting this whole thing. Uber and Meta and I guess Google do it this way now. It takes years of pain to reach to this point. It usually starts with the other kind of "monorepo": The other kind is the "multirepo mon…

There's no good orchestration system that is both easy to implement and has the core features that make a monorepo pleasant to use that is language agnostic.

Bazel is complex and isn't the easiest to pick up for many (though to Google's credit the documentation is getting better). Buck isn't any better in this regard. Pants seems easiest out of all the big ones I've seen but its also a bit quirky, though much easier to get started with in my experience. NX is all over the place in my experience.

Until recently too, most of these monorepo systems didn't have good web ecosystem support and even of those that do they don't handle every build case you want them to, which means you have to extend them in some way and maintain that.

It also doesn't help that most CI systems don't have good defaults for these tools and can be hard to setup properly to take advantage of their advantages (like shared cross machine caching).

As an aside, the best monorepo tooling I have ever used was Rush[0] from Microsoft. If you are working in a frontend / node monorepo or considering it, do take a look. It works great and really makes working in a monorepo extremely uniform and consistent. It does mean doing things 'the rush way' but the trade off is worth it.

[0]: https://rushjs.io

Re: The Ingredients of a Productive Monorepo

#178

Post by zbentley »

dalyons wrote: post #142
zbentley wrote: post #116

Earlier quoted context omitted.

I think that discussions in this area get muddied by people using different definitions of “rapidly”. There are (at least) two kinds of speed WRT tests being run for a large code base. First, there is “rapidly” as pertains to the speed of running tests during development of a change . This is “did I screw up in an obvious way” error checking, and also often “are the tests that I wrote as part of this change passing”…

You are of course entitled to your opinion, and I do appreciate going against the grain, but having worked in an “hours” environment and a “minutes” environment I couldn’t disagree more. The minutes job is so much more pleasant to work with in nearly every way. And ironically ended up being higher quality because you couldn’t lean on a giant integration test suite as a crutch. Automated business metric based canary r…

All very fair points! I think it is perhaps much more situational than I made it out to be, and that functioning in an “hours” environment is only possible as described if some organizational patterns are in place to make it work.

Re: The Ingredients of a Productive Monorepo

#179

Post by senderista »

bigbuppo wrote: post #159

It's kind of weird that both Microsoft and Google were both using Perforce. What does Perforce do that worked well at those companies for so long, and what caused them to dump it? Did they just get tired of the licensing cost? I think what I'm getting at is that maybe the real missing feature isn't whatever it is that allows you to make stupidly large monorepos, but that maybe we should add Perforce's client workspac…

At MSFT we used a Perforce fork (Source Depot), but the Windows codebase was still developed in separate repos ("depots"): kernel, shell, graphics, etc. We had custom tooling to coordinate cross-repo changes, so it was still far from a monorepo.

Re: The Ingredients of a Productive Monorepo

#180

Post by dalyons »

zbentley wrote: post #178
dalyons wrote: post #142

Earlier quoted context omitted.

You are of course entitled to your opinion, and I do appreciate going against the grain, but having worked in an “hours” environment and a “minutes” environment I couldn’t disagree more. The minutes job is so much more pleasant to work with in nearly every way. And ironically ended up being higher quality because you couldn’t lean on a giant integration test suite as a crutch. Automated business metric based canary r…

All very fair points! I think it is perhaps much more situational than I made it out to be, and that functioning in an “hours” environment is only possible as described if some organizational patterns are in place to make it work.

yeah i realized as i wrote that out that my personal conclusions probably don't apply in a monoservice type architecture. If you have a mono(or few) service architecture with a single (or few) db, it is actually feasible to have integration tests that are worth the runtime. The bigger & more distributed you get, the more the costs of integration tests go up (velocity, fragility, maintenance, burden of mirroring production config) and the equation doesnt pencil out anymore. Probably other scenarios where im wrong also.
Post reply on HN