Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Ingredients of a Productive Monorepo

blog.swgillespie.me

Post reply on HN
Print view View original

181–190 of 268 posts

Re: The Ingredients of a Productive Monorepo

#181

Post by senderista »

bluGill wrote: post #132
eddd-ddde wrote: post #127

Earlier quoted context omitted.

This just creates tons of fragmentation. The second you have multiple teams depending on multiple versions you are doomed. You are stuck maintaining multiple versions, with their own quirks and bugs. I think the one version rule is the most important part for a healthy monorepo.

I think one version is important for a healty polyrepo as well. You have to set lines where you say no new features unless you are all up to date. You can allow bug fix only releases to stay behind, but if you write a new feture it must be against the current latest of everything. Otherwise you are doomed because there are so many different versions of everything in use. Some day a zero-day issue will hit all your pr…

Welcome to Amazon.

Re: The Ingredients of a Productive Monorepo

#182

Post by eddd-ddde »

nssnsjsjsjs wrote: post #51

> Any operation over your repository that needs to be fast must be O(change) and not O(repo). This is a good thought! It actually needs to be O(1/commit rate) though, so that having the monorepo doesn't create long queues of commits. Or have some process batch passing ready to merge PRs into a combined PR and try to merge that. And best guess on the failing PR if it fails.

Just make presubmit a fraction of your postsubmit. Each change has fast operations while still having global testing.

Then if postsubmit fails you just have to rerun the intersection of failing tests and affected tests on each change since the last green commit.

Re: The Ingredients of a Productive Monorepo

#183

Post by eddd-ddde »

lenkite wrote: post #37
AlotOfReading wrote: post #18

Earlier quoted context omitted.

One of the big advantages is visibility. You can be aware of what other people are doing because you can see it. They'll naturally come talk to you (or vice versa) if they discover issues or want to use it. It also makes it much easier to detect breakages/incompatibilities between changes, since the state of the "code universe" is effectively atomic.

Not sure if I get it. If you are using a product like Github Enterprise, you are already quite aware of what other people are doing. You have a lot of visibility, source-code search, etc. If you have a CICD that auto-creates issues you already can detect breakages, incompatibilities, etc. State of the "code universe" being atomic seems like a single point of failure.

Imagine team A vendors into their repo team B's code and starts adding their own little patches.

Team B has no idea this is happening, as they only review code in repo B.

Soon enough team A stops updating their dependency, and now you have two completely different libraries doing the "same" thing.

Alternatively, team A simple pins their dependency to team B's repo at hash 12345, then just, never updates... How is team B going to catch bugs that their HEAD introduces on team A's repo?

Re: The Ingredients of a Productive Monorepo

#184

Post by gengstrand »

bob1029 wrote: post #62

This thread is reminding me of a prior one about complexity merchants. I am seeing a lot of sentiment that there is somehow a technical sacrifice by moving to a monorepo. This is absolutely ludicrous unless you fail to grasp the power of a hierarchical file system. I don't see how a big mess like CI/CD is made easier by spreading it out to more points of configuration. To me the whole point of a monorepo is atomic co…

It is true that there are significant benefits to monorepo but it comes at a cost. Managing a monorepo is more expensive than polyrepo. For the details behind that claim, check out https://www.exploravention.com/blogs/monorepo/

The question is this. Do the costs of monorepo justify the benefits for your situation? The answer is not always yes.

Re: The Ingredients of a Productive Monorepo

#185

Post by eddd-ddde »

jbverschoor wrote: post #24

Is there a way to set permissions on certain directories / force partial clones. Not just a sparse clone.

Thats sounds like the type of workflow you would use with gerrit. It's technically multiple repos, but still implements the idea of an atomic build across all repos.

Then you configure ACLs for every repo or branch.

Re: The Ingredients of a Productive Monorepo

#186

Post by teitoklien »

bittermandel wrote: post #73

I firmly believe that us at Molnett(serverless cloud) going for a strict monorepo built with Bazel has been paramount to us being able to make the platform with a small team of ~1.5 full-time engineers. We can start the entire platform, Kubernetes operators and all, locally on our laptops using Tilt + Bazel + Kind. This works on both Mac and Linux. This means we can validate essentially all functionality, even our Bo…

we run everything under systemd managed services instead of k8s and deploy via ansible playbooks at our company, and we similarly use tmuxinator to spin up all the backend api, search engine, databases like qdrant, meilisearch, etc and frontend services all in dev mode with all the terminals auto opening in window panes inside a tmux shell.

It really makes development in dev mode super simple and easy, and running all of the services in local dev environment is as simple as running one command, ‘tmuxinator’ at root of our monorepo and boom everything is up.

Monorepo truly outcompete individual repos for almost all projects, its far more pleasurable ever since I changed to this method of development.

Re: The Ingredients of a Productive Monorepo

#187

Post by amelius »

lolinder wrote: post #110
amelius wrote: post #88

Earlier quoted context omitted.

This will start to become a problem if the stacks need to communicate with each other using versioned protocols.

Why can't you just use versioning in your external-to-the-monorepo APIs and use HEAD within the monorepo? Nothing about combining some projects into a monorepo forces you into dropping everything else we know about software release cycles.

The point is that it is more work.

Re: The Ingredients of a Productive Monorepo

#188

Post by jyounker »

bluGill wrote: post #129
cmrdporcupine wrote: post #103

Earlier quoted context omitted.

The push to fragmentation and atomism is so strong with this generation of devs. The obsession with microservices, dozens of small repositories, splitting everything up from fear of "monoliths." What they're doing is creating a mass of complexity that is turning org-chart problems into future technical ones and at the same time not recognizing the intrinsic internal dependencies of the software systems they're buildi…

That push to fragmentation is in large part because of hard lessons learned from the problems of a monolith. The answer is IMO somewhere in between. Microservices can get too tiny and thus the system becomes impossible to understand. However a monolith is impossible to understand as well. The real problem is you need good upfront architecture to figure out how the whole system fits together. However that is really ha…

Monolith != Monorepo. They're independent concepts.

Re: The Ingredients of a Productive Monorepo

#189

Post by nitwit005 »

CamouflagedKiwi wrote: post #45
AlotOfReading wrote: post #14

One thing I don't usually see discussed in monorepo vs multi repo discussions is there's an inverse Conway's law that happens: choosing one or the other will affect the structure of your organization and the way it solves problems. Monorepos tend to invite individual heroics among common infrastructure teams, for example. Because there are so many changes going in at once, anything touching a common area has a huge n…

That's an optimistic take on what happens in the polyrepo setup. A common alternative (I suspect by far the more common one) is that changes are made to the common area but not propagated to downstream repos, which all end up pinned to different versions of the common repo and struggle to update once they get ~years out of date.

I'd caution that a monorepo isn't a full fix to that. People often make multiple versions of libraries. You have separate 2.X and 3.X versions, with independent source code (or branches), and ask people to migrate to the new one.

There's not really a way around that when you need some behavioral change for the code using the library.

Re: The Ingredients of a Productive Monorepo

#190

Post by l5870uoo9y »

Separating out the database layer in a monorepo package was the best architectural decision I made this year. Now it is my default because at some point you either want to rebuild the existing app entirely or separate out services such as public API access that all need access to the same database.
Post reply on HN