Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepos: Please don’t

medium.com

Post reply on HN
Print view View original

311–320 of 402 posts

Re: Monorepos: Please don’t

#311

Post by joshuamorton »

swish_bob wrote: post #260
thedufer wrote: post #233

Earlier quoted context omitted.

A polyrepo setup at Google's scale would pretty obviously require some dev work. For example, their CI/build story would be way more complex.

While that may be true, I'm not convinced it is a given. Any complicated enough monorepo requires complex CI/build tools, and Bazel/Blaze exist for a reason ...

At Google scale, you'd either need tooling for automated version bumps, or some other infra to manage versioning.

You'd need cross repo bisection.

You'd need a way to run all tests in all repos reflecting a new change.

There's 10s or hundreds more frs I could list.

Re: Monorepos: Please don’t

#312

Post by artemave »

bad_user wrote: post #109
curtis wrote: post #32

My advice is that if components need to release together, then they ought to be in the same repo. I'd probably go further and say that if you just think components might need to release together then they should go in the same repo, because you can in fact pretty easily manage projects with different release schedules from the same repo if you really need to. On the other hand if you've got a whole bunch of component…

Components might need to be released “together”, but if they are worked on by different teams, it means they’ll have a different release process, as in different timeline, different priorities. First of all this is normal, because otherwise the development doesn’t scale. In such a case the monorepo starts to suck. And that’s the problem with your philosophy ... it matters less how the components connect, it matters m…

> Components might need to be released “together”, but if they are worked on by different teams, it means they’ll have a different release process, as in different timeline, different priorities.

released "together" == part of the same feature. Timelines, release process and team priorities are all there to help to deliver features. If they stand in the way, they need to be adjusted. Not the other way around.

Multi repos encourage silos. Silos encourage focusing on the goals of the silo and discourage poking around the bigger picture. Couple that with scrum, that conveniently substitute real progress metrics with meaningless points, and soon enough you end up with an IT department, full on with processes but light on delivering value.

Re: Monorepos: Please don’t

#313

Post by sangnoir »

notacoward wrote: post #300
sangnoir wrote: post #291

Earlier quoted context omitted.

Number of companies is a good metric, because companies own the repos and if it becomes a pain-point, only the developers working at that point in time will be hit by this. Anyone who leaves before this inflection point or joins after it's been solved will not be hit, so I don't think the percentage of developers in that intersection is large.

> after it's been solved I think a quick perusal of this page will show that it's not really "solved" after all. A far higher percentage of developers continue to be affected by large-repo issues than a Python-specific issue (currently #1 story on the front page) or anything to do with Ethereum (currently #7). Are those "horseshit" topics too?

I agree, it's not really solved, but solved "enough". You can't have your cake and eat it, there are tradeoffs involved- if you grow large enough to hit monorepo limitations, you are large enough to invest in tooling that manage your workflow (the tradeoff). However, if you're a small organization, you can't afford the tooling and you're wasting time/quality coordinating polyrepo releases, so you are better off with a monorepo.

> A far higher percentage of developers continue to be affected by large-repo issues than...

Are you suggesting that the results of the HN ranking algorithm at this very moment in time is a good metric of measuring what affects developers? I don't agree, and besides @yowlingcat's opinion that the article is "horseshit" is unrelated to how well its ranked on HN.

Re: Monorepos: Please don’t

#314

Post by mcguire »

yowlingcat wrote: post #269
drugme wrote: post #264

Earlier quoted context omitted.

I found it to be neither horseshit nor irresponsible. A bit overdrawn and skewed in some of its arguments, perhaps. But then again... so was your critique. For example: We had to migrate a polyrepo to a monorepo and it was not fun because it was a migration that should have never had to be done in the first place s/polyrepo/monorepo/ in the above and you have an assertion of about equal plausibility and weight.

No, it is horseshit. 99% of companies will never hit big company VCS scaling issues, and once they do, they're on their own. To characterize that scale as common is one of the most embarrassing failures of modern software engineering. People are so embarrassed to use well worn tooling and accept that large scale is both uncommon and something that doesn't invalidate tried and true patterns for smaller scales. It's ut…

You realize many, if not most, people reading this work at places already big enough to have "VCS scaling issues". I've seen more than a few monorepos, but I've never seen one used as anything but a collection of small repos.

Re: Monorepos: Please don’t

#315

Post by bluGill »

swiftcoder wrote: post #286
oblio wrote: post #174

Earlier quoted context omitted.

> You just need a handful of developers working on the infra to make it happen. With thousands of developers banging on the code base, it's going to be more than "a handful of developers". It's going to be at least a few "handfuls" of developers full time plus probably many, many other full time equivalents spread out throughout the whole user base (testing, supporting other users, etc.).

In reality, it's several hundred developers working full time on infra, and they are all overworked, and gradually falling behind. Monorepos at the scale of Google/Facebook are hard . We aren't talking about maintaining Mercurial here - we are talking about developing a brand new distributed VCS that happens to be 'Mercurial-compatible', and deploying/maintaining it for tens of thousands of developers working simulta…

Development with thousands of developers is hard. The problems with monorepo and polyrepo are only subtly different. Either way you need a fairly large team just to handle the tools you need to solve your problems. Some of your problems will be because of repo organization (again, both choices have downsides that you need custom tooling to solve).

Note that most of your problems will be related to having a thousands of developers and repo organization is irrelevant.

Re: Monorepos: Please don’t

#316

Post by andrewprock »

gitgud wrote: post #240
forkerenok wrote: post #154

Earlier quoted context omitted.

And no, you cannot split a monorepo into a polyrepo easily. Been there, done that. _The reason is that working in a monorepo versus multiple repos influences the architecture quite a lot and the monorepo leads to very unclear boundaries. I think you are conflating a monorepo (where boundaries can still be established, e.g. via a module isolation mechanism specific to the stack used) with a "monoproject"/"monomodule",…

If the projects within the monorepo are decoupled and have clear boundaries then why not have them in separate repositories?... In my opinion monorepos make refactoring dependant projects much easier. However it is much harder to establish and enforce clear boundaries...

With monorepos you don't have to manage PRs for 8 different repositories when adding a feature.

In my experience it's hard to establish clear boundaries, regardless of repository kind. It may be more difficult to create features which are tightly coupled across multiple repositories, but people do it regularly. And when they do, you suddenly have to manage and maintain synced features across multiple repositories.

In fact, the repo tool for the android project makes it quite easy to develop features across repositories, thus lowering the boundaries significantly.

Re: Monorepos: Please don’t

#317

Post by curtis »

thanatos_dem wrote: post #289
yowlingcat wrote: post #256

I think this article is complete horseshit. A monorepo will serve you 99% of the time until you hit a certain level of scale when you get to worry about whether a monorepo or a polyrepo is actually material. Most cases are never going to get there. Before that point, a polyrepo is purely a distraction and makes synchronous deployment really painful. We had to migrate a polyrepo to a monorepo and it was not fun becaus…

I work on CI/CD systems, and that’s one thing that definitely gets harder in a monorepo. So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy? It doesn’t take a large amount of scale to make rebuilding and retesting everything impossible. In a poly repo world, the repository is generally the unit of building and deployment. In monorepo it gets more messy. For inst…

> So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy?

If you're using Git, then typically for each push to the remote repository you get a notification with this data in it:

  BRANCH        # the remote branch getting updated
  OLD_COMMIT    # the commit the branch ref was pointing to before the push
  NEW_COMMIT    # the commit the branch ref was pointing to after the push

  # To get the list of files that changed in the push:
  git diff --name-only "$OLD_COMMIT" "$NEW_COMMIT"
Once you know which files changed in a push you can figure out which artifacts you need to build. Right now you'll have to write that tooling yourself since I don't know of any off-the-shelf tools that do it. In my company's case, we have "project.yml" files scattered through the repo telling us which directories have buildable artifacts and what branches each one needs to be built for. The tooling to support this is a few hundred lines of Bash and Python. In our case we're still small enough that we can brute force some stuff, but we can easily improve the tooling as we go along.

Re: Monorepos: Please don’t

#318

Post by yoz-y »

ryanbrunner wrote: post #214
tigershark wrote: post #185

Earlier quoted context omitted.

I still can’t see how you would have a shared library for a C# gui and a Java server for example. Your communication layer would obviously live in both repositories. Even in case you are using the same language and you do have shared libraries then what is the problem? The shared libraries would surely be shared with other projects so it makes sense to have them in a separate repository.

In cases where there's a high degree of churn (i.e. early-stage startups) in shared libraries, updating those libraries can cause a large amount of busywork and ceremony. If you had a `foo()` function shared between the GUI and the server (or two services on your backend, or whatever), in a monorepo your workflow is: - Update foo() - Merge to master - Deploy In a polyrepo where foo() is defined in a versioned, shared…

Some package managers that support git repos as dependency versions can offset this in development.

Re: Monorepos: Please don’t

#319

Post by tigershark »

ryanbrunner wrote: post #214
tigershark wrote: post #185

Earlier quoted context omitted.

I still can’t see how you would have a shared library for a C# gui and a Java server for example. Your communication layer would obviously live in both repositories. Even in case you are using the same language and you do have shared libraries then what is the problem? The shared libraries would surely be shared with other projects so it makes sense to have them in a separate repository.

In cases where there's a high degree of churn (i.e. early-stage startups) in shared libraries, updating those libraries can cause a large amount of busywork and ceremony. If you had a `foo()` function shared between the GUI and the server (or two services on your backend, or whatever), in a monorepo your workflow is: - Update foo() - Merge to master - Deploy In a polyrepo where foo() is defined in a versioned, shared…

It seems to me that the real problem is your toolchain. In a previous project the workflow was like this:

Update foo() Merge to master Publish shared library Deploy

So as you can see the only step added was to publish the shared library that would automatically update the version in all the projects using it. If you are really doing everything manually I can understand that this is a pain, but this has nothing to do with the monorepo / multiple repo distinction, this is a tooling problem.

Re: Monorepos: Please don’t

#320

Post by jskaggz »

yowlingcat wrote: post #256

I think this article is complete horseshit. A monorepo will serve you 99% of the time until you hit a certain level of scale when you get to worry about whether a monorepo or a polyrepo is actually material. Most cases are never going to get there. Before that point, a polyrepo is purely a distraction and makes synchronous deployment really painful. We had to migrate a polyrepo to a monorepo and it was not fun becaus…

Here here yowlingcat. Article is a way too prescriptive and agreed, borders on irresponsible. Monorepo vs polyrepo argument is way too broad a subject to create generalized stereotypes like this. These opinions sadly are taken as facts by impressionable managers, new developers, etc, and have cascading effects on the rest of us in the industry. Use what makes sense in the project environment and team, don't just throw shade at teams who are successfully and productively using monorepos where they make sense. Sure there is good reason to split things up on boundaries sometimes, (breaking out libraries, rpc modules, splitting along dev team boundaries, etc etc etc), but not blindly by default. Will Torvalds split up the kernel into a polyrepo after reading this article? Something tells me that would be a bit disruptive.
Post reply on HN