Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

201–210 of 268 posts

Re: The Ingredients of a Productive Monorepo

#201

Post by KaiserPro »

DrScientist wrote: post #93
KaiserPro wrote: post #87

One of the things not covered here is how to deal with versioning. By default a monorepo will give you $current and nothing else. A monorepo is not a bad idea, but you should think about either preventing breaking changes in some dependency killing the build globally, or have some sort of artefact store that allows versioned libraries (both have problems, you'll need to work out which is better for you. )

I think a key idea often associated with the use of a monorepo is to encourage developer behaviour to do the integration/mitigation work at the point of change, rather than creating lots of integration debt in the form of versions ( however you do it ). You need to look at your development model as a whole and decide whether the happy path incentivises good or bad development practices. Do you want to incentivise the…

I worked at a startup with a "monorepo" (C++, cuda and python) it worked well and wasn't too hard to manage. Once someone bit the bullet and made some robust bazel spells it was brilliant to use and multi-platform too.

Worked at a FAANG with a monorepo, and everything was partially broken most of the time. Its trivial to bring in dependencies, which is great, super fast re-use.

The problem is, its trivial to add dependencies. That means that bringing in a library to manage messages also somehow requires a large amount of CUDA code as well.

A basic python programme would endup having something like >10k build items to go through each build.

Re: The Ingredients of a Productive Monorepo

#202

Post by marcosdumay »

bluGill wrote: post #100
spankalee wrote: post #54

Earlier quoted context omitted.

The whole point of a monorepo is to force you to update all of the consumers, and to realize that breaking changes are expensive. The two monorepo ways to do this: 1. Use automated refactoring tools that now work because it's one repo 2. Add the new behavior, migrate incrementally, then remove the old behavior

Both of those work in polyrepo. You need a tools team to make it happen though, just like monorepo needs a tool team. The tools needed are different but you still need them.

With enough tooling, a monorepo or a polyrepo environment look exactly the same. Those articles are "Look. This is a good way to organize your code", not something that tells you one of those is better than the other.

Re: The Ingredients of a Productive Monorepo

#203

Post by munksbeer »

eadmund wrote: post #89
bittermandel wrote: post #73

I firmly believe that us at Molnett(serverless cloud) going for a strict monorepo built with Bazel has been paramount to us being able to make the platform with a small team of ~1.5 full-time engineers. We can start the entire platform, Kubernetes operators and all, locally on our laptops using Tilt + Bazel + Kind. This works on both Mac and Linux. This means we can validate essentially all functionality, even our Bo…

> a small team of ~1.5 full-time engineers Yes, with one and a half FTEs you should only have a single repo. My experience with Bazel has been extremely bad, but I don’t think that it should necessarily be avoided completely. It may actually have some value on extremely large multi-team projects. But for less than two FTEs it seems like massive overkill. I believe that you could do what you need with Kind (and maybe…

> My experience with Bazel has been extremely bad

Would you mind elaborating and providing some examples of what was bad?

We have a monorepo built using bazel, and at first when new to bazel, I was pretty frustrated. But now I can't think of any issue I've had with it recently.

But we do have a relatively simple setup.

Re: The Ingredients of a Productive Monorepo

#204

Post by jayd16 »

bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

Here's the issue... Being able to track commits across multiple projects is just a nice to have. Does it even significantly increase your ability to track dependencies or trigger down stream tests? You can track that stuff with multi-repo automations. Maybe it helps a bit but it's not free and its not complete.

Your deploys don't become atomic. Your builds don't become atomic. At best you get to tell yourself you can version a bit more loosely.

Mono-repos themselves do not scale easily. Inherently its a harder technical problem to solve. You need to toss git and find something better, which is not easy. Its work. Its so much work that it is incredibly clear that you've never experienced it yourself.

Re: The Ingredients of a Productive Monorepo

#205

Post by spankalee »

marcosdumay wrote: post #202
bluGill wrote: post #100

Earlier quoted context omitted.

Both of those work in polyrepo. You need a tools team to make it happen though, just like monorepo needs a tool team. The tools needed are different but you still need them.

With enough tooling, a monorepo or a polyrepo environment look exactly the same. Those articles are "Look. This is a good way to organize your code", not something that tells you one of those is better than the other.

Most monorepos imply that all first-party code only available at one version. Polyrepos usually allow first-party code to depend on old versions of other first-party code.

Re: The Ingredients of a Productive Monorepo

#206

Post by datadrivenangel »

jyounker wrote: post #188
bluGill wrote: post #129

Earlier quoted context omitted.

That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith. The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well. The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really ha…

Monolith != Monorepo. They're independent concepts.

The tooling defaults around Github encourage having one thing per repo.

Re: The Ingredients of a Productive Monorepo

#207

Post by cmrdporcupine »

bluGill wrote: post #129
cmrdporcupine wrote: post #103

Earlier quoted context omitted.

The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're buildi…

That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith. The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well. The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really ha…

https://askastaffengineer.com/

Re: The Ingredients of a Productive Monorepo

#208

Post by WorldMaker »

CamouflagedKiwi wrote: post #46
shawabawa3 wrote: post #19

Earlier quoted context omitted.

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

I think one reason is that there are various big companies (Google, Microsoft, Meta) who have talked about the tech they've deployed to make monorepos work, but I at least have never seen an equivalent big successful company describe their polyrepo setup, how they solved the pain points and what the tech around it looks like.

I also think a lot of it is quiet for a reason. There aren't interesting problems to solve. A lot of it is boring. It isn't without pain, but most of the pain consists of lots of little papercuts rather than big giant showstopping injuries. A lot of the papercuts are just annoying enough itches that aren't worth scratching. Or are solved with ecosystems of normal, boring tools like Jenkins or GitHub Advanced Security or SonarQube or GitHub Actions or… Boring off-the-shelf tools for boring off-the-shelf pain points.

Re: The Ingredients of a Productive Monorepo

#209

Post by crazygringo »

yencabulator wrote: post #191
bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

> To me the whole point of a monorepo is atomic commits for the whole org. The belief that a monorepo makes a change somehow more atomic is one of the traps. From the article: > The greatest power and biggest lie of the monorepo is that it is possible to make atomic commits across your entire codebase. [...] > Your monorepo now contains many different deployable artifacts that deploy at different times. It is also te…

That's confusing two different things, though.

A monorepo does make changes atomic in the code. There's no trap there.

You're talking about deployment, and yes when deployment is staggered, then obviously all atomic changes need to be backward-compatible, or else be very carefully orchestrated. But that doesn't have anything to do with monorepo vs polyrepo. That's just staggered deployment.

You have to deal with backwards compatibility in both cases. But at least with the monorepo you can see and track and merge all the changes related to a feature in one place, and be able to roll them back in one place.

Re: The Ingredients of a Productive Monorepo

#210

Post by cloogshicer »

bluGill wrote: post #101
cloogshicer wrote: post #67

Earlier quoted context omitted.

Thanks for the info! Seems like a big restriction to me.

it is the only sane thing to do. Allowing everyone to use their own fork means when a major bug is found you have to fix thousands of forks. If the bug is a security zero day you don't have time.

Couldn't you just leave the other consumer at the old release (presumably well tested, stable)?

I don't see how being forced to upgrade all consumers is a good thing.

Post reply on HN