Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

161–170 of 268 posts

Re: The Ingredients of a Productive Monorepo

#161

Post by dezgeg »

bluGill wrote: post #139
chrismatic wrote: post #76

The point about trying to stick with a single language build tooling really cannot be stressed enough. It is what prompted me to write a simplified version of Bazel, a generic "target determinator" with caching capabilities if you will. I call it "Grog", the monorepo build tool for the grug-brained developer. https://grog.build/why-grog/

If a single language is an option you are a small project that is not facing the problems people on large projects are facing. A monorepo will be easy for you without read the article and the lessons learned. Come back when you have millions of lines of code, written over decades by hundreds (or thousands) of full time developers.

What a weird take, "millions of lines of code, written over decades" applies to quite many C (or C++) codebases where using a high-level language is not a possibility (and companies that do have such codebases are pretty conservative and don't even talk about Rust no matter how great fit it would be).

Re: The Ingredients of a Productive Monorepo

#162

Post by krschultz »

cloogshicer wrote: post #47

Here's what I never got about monorepos: Imagine you have an internal library and also two consumers of that library in the repo. But then you make breaking changes to the library but you only have time to update one of the consumers. Now how can the other consumer still use the old version of that library?

You don't make breaking changes. You provide the new API and the old API at the same time, and absorb the additional complexity as the library owner. Best case scenario everyone migrates to the new API and eventually remove the old one. This sounds onerous, but keep in mind at a certain scale there is no one commit in production at any given time. You could never roll out an atomic breaking change anyway, so going through this process is a reflection of the actual complexity involved.

Re: The Ingredients of a Productive Monorepo

#163

Post by kccqzy »

spankalee wrote: post #57

I love monorepos, but in large organizations they have a counter-intuitive incentive for teams to _not_ allow other teams to depend on them, which can _reduce_ code reuse - the opposite of what some adopters want. This issue is that users of a library can put almost infinite friction on the library. If the library team wants to make a change, they have to update all the use sites, but Hyrum's Law will get you because…

At least the benefit of a monorepo is that you can find all the use sites in the first place and correct these wrong uses. You can even correct them atomically if you so wish.

Re: The Ingredients of a Productive Monorepo

#164

Post by lenkite »

jeffbee wrote: post #135
lenkite wrote: post #37

Earlier quoted context omitted.

Not sure if I get it. If you are using a product like Github Enterprise, you are already quite aware of what other people are doing. You have a lot of visibility, source-code search, etc. If you have a CICD that auto-creates issues you already can detect breakages, incompatibilities, etc. State of the "code universe" being atomic seems like a single point of failure.

GitHub search is insanely bad and it cannot do things like navigating to definitions between repos in an org.

If you want code search and navigation over a closed subgraph of projects that build into an artifact - opengrok does the job reasonably well.

Re: The Ingredients of a Productive Monorepo

#165

Post by kccqzy »

bluGill wrote: post #152
eddd-ddde wrote: post #134

Earlier quoted context omitted.

In a polyrepo it is more common that the update simply happens, now repo A depends on v1 and repo B depends on v2, then a year has passed and repo A doesn't even remember they still depend on an old insecure library.

That is a downside of a polyreop that you will need to figure out how to mitigate. It doesn't matter if you go monorepo or polyrepo you wil have issues as your project grows. You will need to mitigate those issues somehow.

In a polyrepo it is common to say I depend on this specific git SHA of that other repo. In a monorepo it is weird and unheard of to say I depend on this specific SHA of the current repo. It's a matter of defaults.

Re: The Ingredients of a Productive Monorepo

#166

Post by lenkite »

anon7000 wrote: post #69
zvr wrote: post #58

Genuine question, because I've never worked somewhere with a monorepo infrastructure: is it really "one repo for all code in the organization" or "one repo for everything related"? In my organization we have around 70k internal git repos (and an order of magnitude fewer public ones), but of course not everything is related to everything else; we produce many distinct software products. I can understand "collect every…

When you have N repos, you also have N ways of managing dependencies, N ways of doing local bin scripts and dev environment setups, N projects with various out of date & deprecated setups, N places to look when you need to upgrade a vulnerable dependency, N services which may or may not configure telemetry in a consistent way, N different CI & deployment workflows… It just gets very difficult to manage, especially if…

> When you have N repos, you also have N ways of managing dependencies, N ways of doing local bin scripts and dev environment setups, N projects with various out of date & deprecated setups, N places to look when you need to upgrade a vulnerable dependency, N services which may or may not configure telemetry in a consistent way, N different CI & deployment workflows…

No, you do not, unless you mean N=1. Build scripts/tooling/linters etc are put into a different repo, are released and consumed by each individual repo.

Re: The Ingredients of a Productive Monorepo

#167

Post by jonthepirate »

I'm on the build team at DoorDash. We're in year 1 of our Bazel monorepo journey. We are heavy into Go, already have remote execution and caching working, and are looking to add support for Python & C++ soon.

If this sort of stuff happens to be something you might want to work on, our team has multiple openings... if you search for "bazel" on our careers page, you'll find them.

Re: The Ingredients of a Productive Monorepo

#168

Post by ec109685 »

spankalee wrote: post #57

I love monorepos, but in large organizations they have a counter-intuitive incentive for teams to _not_ allow other teams to depend on them, which can _reduce_ code reuse - the opposite of what some adopters want. This issue is that users of a library can put almost infinite friction on the library. If the library team wants to make a change, they have to update all the use sites, but Hyrum's Law will get you because…

I would still say code is more likely to be reused in the monorepo versus trying to take an external dependency in the poly repo case. Just the ease of making a change to target your case is so much higher.

Re: The Ingredients of a Productive Monorepo

#169

Post by stackskipton »

CamouflagedKiwi wrote: post #46
shawabawa3 wrote: post #19

Earlier quoted context omitted.

Every time I've been monorepos compares with polyrepos it's always "monorepo plus millions of dollars of custom tool engineering" vs "stock polyrepo" Why can't we add millions of dollars of tool engineering on top of polyrepos to get some of the benefits of monorepos without a lot of the pain? E.g. it wouldn't be too hard to create "linked" PRs across repos for changes that span projects, with linked testing infrastr…

I think one reason is that there are various big companies (Google, Microsoft, Meta) who have talked about the tech they've deployed to make monorepos work, but I at least have never seen an equivalent big successful company describe their polyrepo setup, how they solved the pain points and what the tech around it looks like.

>equivalent big successful company describe their polyrepo setup, how they solved the pain points and what the tech around it looks like.

I've worked at big successful F500 boring companies with polyrepo setup and it's boring as well. For this company, it was Jenkins checked out the repo, ran the Jenkins file, artifact was created and stuck into JFrog Artifactory. We would update Puppet file in our repo and during approved deploy window in ServiceNow, Puppet would do the deploy. Because of this, Repos had certain fixed structure which was annoying at times.

Pain Points that were not solved is 4 different teams involved in touching everything (Jenkins, Puppet, InfoSec and dev team) and break downs that would happen.

Post reply on HN