Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monorepoize – Bash scripts for creating a monorepo out of smaller repos

github.com

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#71

Post by mumblemumble »

pierrebai wrote: post #63
mumblemumble wrote: post #57

Earlier quoted context omitted.

I worked in a company that had at least 100 repos, often with interdependencies, and was managing it well using svn. Then we migrated to Git, which doesn't have a good equivalent to svn's externals, and things kind of went to heck for a while. We eventually hacked together an in-house system that brought back some of what we were missing, but it never worked as smoothly as svn externals did. I was originally one of t…

git has submodules. A repo can thus host any number of sub-repos. Using modules can also be pretty much like having a monorepo if you have a master repo with all others under it. This way, you can still publish things like public API as a repo.

They don't work quite the same way as svn externals, though, in a way that didn't work for us. In particular, they leave no good way to deal with merging in upstream changes in a submodule's code if you've also made some of your own (provisional) edits.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#72

Post by mceachen »

pierrebai wrote: post #63
mumblemumble wrote: post #57

Earlier quoted context omitted.

I worked in a company that had at least 100 repos, often with interdependencies, and was managing it well using svn. Then we migrated to Git, which doesn't have a good equivalent to svn's externals, and things kind of went to heck for a while. We eventually hacked together an in-house system that brought back some of what we were missing, but it never worked as smoothly as svn externals did. I was originally one of t…

git has submodules. A repo can thus host any number of sub-repos. Using modules can also be pretty much like having a monorepo if you have a master repo with all others under it. This way, you can still publish things like public API as a repo.

Submodules don't allow for atomic commits that span repos, though. Atomic cross-project commits are the most compelling selling feature for monorepos IMHO.

(A common case for this is changing a function's API, and fixing all downstream consumers, in one tidy, holistic PR).

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#73

Post by cryptonector »

frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

The biggest barrier to monorepos is size, which is a problem for `git log` and related (e.g., `git blame`). The Microsoft work on Git will make a lot of that trouble go away, but I suppose there will always be some issues to working with a huge monorepo.

Q: But what's the alternative? A: A repo forest, or a repo web.

Well, repo forests/webs have similar issues crop up anyways. You could say that whether a mega-codebase is spread over a mega-monorepo or a mega-repoforest doesn't change the fact that it's a mega-codebase -- big comes with problems no matter what.

And many enterprises can't help getting to have megacodebases. Operating systems (including distros in the Linux case) are huge. So are ecosystems for various popular programming languages. Enterprise apps easily add up to more in large enterprises.

So once you're dealing with mega-codebases... if you can find a mega-monorepo that works, that's probably where you want to be.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#74

Post by foobarian »

pjmlp wrote: post #52
wikibob wrote: post #38

Earlier quoted context omitted.

https://www.google.com/search?q=microsoft+monorepo https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

That doesn't say that everything related to Windows is developed on the same repo, just the kernel and several core components.

How did I miss this! Can you imagine someone claiming Windows is going to be in git as few as 10 years ago? This world never ceases to amaze me. What happened to SourceSafe?

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#75

Post by throwaway894345 »

bluGill wrote: post #45
auxym wrote: post #26

Earlier quoted context omitted.

I worked in a company where a million line project was split in about 130 repos. Features or bug fixes frequently required syncing PRs between multiple repos. We needed scripts to create feature branches in all repos, or bump up the version everywhere. 4 or 5 repos would probably have been sane. 130 was hell to manage.

If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your a…

Even with the best architecture, you probably have some shared libraries that are used across teams. These will often need to be updated, and they will affect several other packages (other repos, in the multirepo case). The only way to avoid this is to duplicate all code. Moreover, you can have a great architecture, but your teams are "feature-oriented" instead of service oriented (or some other vertically oriented team structure) in which case they'll need to touch many different repos (I don't imagine feature-oriented repos are a good idea) per ticket.

I'm increasingly convinced that the monorepo approach is the right one, but the tooling needs to get better. Bazel is a hydra of incidental complexity and Nix has the right core idea but suffers from a variety of other problems (poor documentation, gratuitously and unnecessarily unfamiliar, too few escape hatches, etc). These issues aren't fundamental to the domain, and there's no reason these tools couldn't change course nor is there a reason that another tool couldn't come in and eat their lunch.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#77

Post by beagle3 »

ratww wrote: post #70
beagle3 wrote: post #68

Earlier quoted context omitted.

> In my experience most engineers have the best intentions and want to do the best work possible I agree, most of them do - but I've also been in a place that highers the best of the best, and despite the best intentions, some people cannot put their ego aside and the overall result is between less-than-ideal and downright-horrible

> ...until you force them into working in a bad environment where he's treated like... a cog in the machine with zero ability to influence his own productivity, happiness or even the quality of his code.

I disagree (unless your "bad environment" gets tautologically redefined to mean any place where people with good intentions don't get good results.... which you might).

I was working in a place that employed incredibly smart and capable but highly opinionated people, and did not treat them as cogs, and did let them influence their productivity, happiness, and quality of the code.

However, there was a major philosophical disagreement between the "functional programming" camp and the "object oriented" camp (with a minor "procedural programming" camp), and designs occasionally became "I am smarter than you" contests, the overall system being extremely incoherent because every side wss able to influence their own productivity.

It worked, because they were smart. It was not a bad environment for any single person. But it would have been a much better place for everyone if there was a benevolent dictator (but nevertheless, dictator) who enforced one way or the other - the randomly reached middle ground has all of the deficiencies of all approaches, and none of the benefits.

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#78

Post by dkryptr »

neeeeees wrote: post #49
zaphar wrote: post #48

The single biggest impediment to monorepo's might be jenkins. Monorepos need a radically different way to do CI/CD and none of the existing tooling does it well. There is probably a market out there for some tooling to do good Monorepo CI/CD without requiring hacky scripts and polling.

This is really a build system issue - ie figuring out which parts of the (mono)repo are affected by which change. Bazel, for example, depends on explicit, per-directory, BUILD files, and does a fairly good job of finding dependants.

Bazel works great for our pretty simplistic monorepo. My main gripe is that it's a pretty big pain to add support for custom build requirements. There have been times where it takes a whole sprint to implement build functionality because, imo, it's hard to run with unless you've been in the ecosystem for a while to understand everything.

I'm curious though, am I the only one who feels this way?

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#79

Post by juped »

globular-toast wrote: post #76

I'm constantly amazed at how fashions come and go in cycles. Only a few years ago everyone was vigorously splitting everything up into microservices and now the monolithic repository is the latest trend.

Both of those are current trends - microservices in a monorepo - as far as I can tell. (Not a fan of either.)

Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos

#80

Post by bregma »

frollo wrote: post #18

After working on a monorepo and then the split up repos for the same codebase, I cannot fathom why somebody would want to take small repos and merge them in a single repo. The mess and complexity just increase.

I read somewhere on the internet that Google uses a monorepo. I want to be like Google, so I need to use a monorepo.
Post reply on HN