Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

251–260 of 268 posts

Re: The Ingredients of a Productive Monorepo

#251

Post by code_biologist »

cloogshicer wrote: post #210
bluGill wrote: post #101

Earlier quoted context omitted.

it is the only sane thing to do. Allowing everyone to use their own fork means when a major bug is found you have to fix thousands of forks. If the bug is a security zero day you don't have time.

Couldn't you just leave the other consumer at the old release (presumably well tested, stable)? I don't see how being forced to upgrade all consumers is a good thing.

I don't see how being forced to upgrade all consumers is a good thing.

It forces implementers of broad or disruptive API or technical changes to be responsible for the full consequences of those decisions, rather than the consumers of those changes who likely don't have context. People make better choices when they have to deal with the consequences themselves.

It also forces the consequences to be incurred now as opposed to 6 months later when a consumer of the old library tries to upgrade, realizes they can't easily, but they need a capability only in the new version, and the guy who made the API change has left the company for a pay raise elsewhere.

As a plus, these properties enable gigantic changes and migrations with confidence knowing there aren't any code or infrastructure bits you missed, and you're not leaving timebombs for a different project's secret repo that wasn't included in the gigantic migration.

Bluntly, if you can't see why many people like it (even if you disagree), you probably haven't worked in an environment or technical context where mono vs poly truly matters.

Re: The Ingredients of a Productive Monorepo

#252

Post by oftenwrong »

crazygringo wrote: post #209
yencabulator wrote: post #191

Earlier quoted context omitted.

> To me the whole point of a monorepo is atomic commits for the whole org. The belief that a monorepo makes a change somehow more atomic is one of the traps. From the article: > The greatest power and biggest lie of the monorepo is that it is possible to make atomic commits across your entire codebase. [...] > Your monorepo now contains many different deployable artifacts that deploy at different times. It is also te…

That's confusing two different things, though. A monorepo does make changes atomic in the code. There's no trap there. You're talking about deployment, and yes when deployment is staggered, then obviously all atomic changes need to be backward-compatible, or else be very carefully orchestrated. But that doesn't have anything to do with monorepo vs polyrepo. That's just staggered deployment. You have to deal with back…

There is also a question of what developers should be developing against. A typical monorepo approach allows many developers to develop only against trunk, and not have to allow the messy world of releases and deployments into their brain.

Re: The Ingredients of a Productive Monorepo

#253

Post by oftenwrong »

chrismatic wrote: post #76

The point about trying to stick with a single language build tooling really cannot be stressed enough. It is what prompted me to write a simplified version of Bazel, a generic "target determinator" with caching capabilities if you will. I call it "Grog", the monorepo build tool for the grug-brained developer. https://grog.build/why-grog/

I am excited to learn of this project. I started working on something quite similar recently. It's a surprisingly unaddressed niche.

One thing your tool appears to be missing (IMO) is execution sandboxing. This is useful, as you likely know, for avoiding undeclared dependencies and for avoiding dirty builds due to actions polluting the source directory, among other things. I was playing around with allowing configurable sandboxing, with symlink forest and docker as two intial options.

Re: The Ingredients of a Productive Monorepo

#255

Post by DrScientist »

KaiserPro wrote: post #201
DrScientist wrote: post #93

Earlier quoted context omitted.

I think a key idea often associated with the use of a monorepo is to encourage developer behaviour to do the integration/mitigation work at the point of change, rather than creating lots of integration debt in the form of versions ( however you do it ). You need to look at your development model as a whole and decide whether the happy path incentivises good or bad development practices. Do you want to incentivise the…

I worked at a startup with a "monorepo" (C++, cuda and python) it worked well and wasn't too hard to manage. Once someone bit the bullet and made some robust bazel spells it was brilliant to use and multi-platform too. Worked at a FAANG with a monorepo, and everything was partially broken most of the time. Its trivial to bring in dependencies, which is great, super fast re-use. The problem is, its trivial to add depe…

Great point - it's one of my pet peeves - automatic chained dependency management - at the what-do-I-need-to-build-everything level ( which is not the same as what do I need for my particular use ).

I think dependency management should be manual - make it intentional - and yes slightly harder.

If you have a static typed language, ( reflection like mechanisms aside ) you can make the compiler do the work in determining if the right dependencies are there and you can massively cut down the dependencies trees.

ie there is a mismatch between the semantics of automatic dependencies tree's and what you actually need when you import.

So if you need want to use library B from module A - I don't need the dependancies of B such that the whole of B compiles , I just need the dependencies of B that enable the my very specific use of B.

So if you add the module B to your project and run your compiler then it should tell you what further dependencies you need - rather than assuming you need to be in C and because you brought in C you also need D and E etc etc.

Re: The Ingredients of a Productive Monorepo

#256

Post by DrScientist »

DrScientist wrote: post #255
KaiserPro wrote: post #201

Earlier quoted context omitted.

I worked at a startup with a "monorepo" (C++, cuda and python) it worked well and wasn't too hard to manage. Once someone bit the bullet and made some robust bazel spells it was brilliant to use and multi-platform too. Worked at a FAANG with a monorepo, and everything was partially broken most of the time. Its trivial to bring in dependencies, which is great, super fast re-use. The problem is, its trivial to add depe…

Great point - it's one of my pet peeves - automatic chained dependency management - at the what-do-I-need-to- build -everything level ( which is not the same as what do I need for my particular use ). I think dependency management should be manual - make it intentional - and yes slightly harder. If you have a static typed language, ( reflection like mechanisms aside ) you can make the compiler do the work in determin…

If you don't have a compiler ( or have dynamic loading anyway ) then your test becomes does it run, rather than does it compile in terms of finding missing dependencies.

Given you only add dependencies once, I don't think it's a big deal to force developers to spend 5 mins determining exactly what they need rather than importing the world.

Re: The Ingredients of a Productive Monorepo

#257

Post by eddd-ddde »

lenkite wrote: post #199
eddd-ddde wrote: post #183

Earlier quoted context omitted.

Imagine team A vendors into their repo team B's code and starts adding their own little patches. Team B has no idea this is happening, as they only review code in repo B. Soon enough team A stops updating their dependency, and now you have two completely different libraries doing the "same" thing. Alternatively, team A simple pins their dependency to team B's repo at hash 12345, then just, never updates... How is tea…

This is already caught by multi-repo tooling like Github today. If you vendor in an outdated version with security vulnerabilities, issues are automatically raised on your repo. Team B doesn't need to do anything. It is Team-A's responsibility to adopt to latest changes.

Curious because I haven't seen this myself. Do you mean, GitHub detects outdated submodule references? Or, GitHub detects copy of code existing in another repo, and said code has had some patches upstream?

Re: The Ingredients of a Productive Monorepo

#258

Post by MiscCompFacts »

jyounker wrote: post #188
bluGill wrote: post #129

Earlier quoted context omitted.

That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith. The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well. The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really ha…

Monolith != Monorepo. They're independent concepts.

I work at a company that has several micro services and a backend all in one monorepo. For some anecdata.

Re: The Ingredients of a Productive Monorepo

#259

Post by lenkite »

eddd-ddde wrote: post #257
lenkite wrote: post #199

Earlier quoted context omitted.

This is already caught by multi-repo tooling like Github today. If you vendor in an outdated version with security vulnerabilities, issues are automatically raised on your repo. Team B doesn't need to do anything. It is Team-A's responsibility to adopt to latest changes.

Curious because I haven't seen this myself. Do you mean, GitHub detects outdated submodule references? Or, GitHub detects copy of code existing in another repo, and said code has had some patches upstream?

Github has dependabot https://docs.github.com/en/code-security/dependabot/dependab... which can also raises PR's, though your mileage may greatly vary here depending on your language.

You can also configure update of dependencies https://docs.github.com/en/code-security/dependabot/dependab...

These work with vendored dependencies too.

(In our org, we have our own custom Go tool that handles more sophisticated cases like analyzing our divergent forks and upstream commits and raising PR's not just for dependencies, but for features. Only works when upstream refactoring is moderate though)

Re: The Ingredients of a Productive Monorepo

#260

Post by hakfoo »

przmk wrote: post #82
lxe wrote: post #8

So there are 2 kinds of big tech monorepos. One is the kind described in the article here: "THE" monorepo of the (mostly) entire codebase, requiring custom VCS, custom CI, and a team of 200 engineering supporting this whole thing. Uber and Meta and I guess Google do it this way now. It takes years of pain to reach to this point. It usually starts with the other kind of "monorepo": The other kind is the "multirepo mon…

At my current $dayjob, there is a backend that is split into ~11 git repos which results in a single feature being split among 4-5 merge requests and it's very annoying. We're about to begin evaluating monorepos to group them all (among other projects). What would the alternative to a monorepo be in this case, knowing that we can't bundle the repos together?

At mine we ended up with two very comparable webapp products due to an acquisition.

One is built as a monorepo and we have a shared dev server where each user can run their own copies in a home directory.

The other is built as a collection of Docker containers that devs run locally. Nobody from the monorepo team likes dealing with it. Resyncing requires a much more elaborate Git process than a single fetch and pull. A simple task can spawn five merge requests to digest. We have loads of extra effort just making the QA team is in the same place as devs.

If nothing else, there's huge simplification from "I can access your copy of the code base and see the same error you're seeing" without trying to screenshare or remote-desktop.

Post reply on HN