From having done this a couple of times in the last few years: make sure you use the mandatory history rewrite in this process to get rid of history you don't want. Team christmas party videos that got checked into master (just kidding) can easily be removed before switching.
Monorepoize – Bash scripts for creating a monorepo out of smaller repos
91–100 of 103 posts
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#92Earlier quoted context omitted.
I feel like the library needed is actually the opposite. Take a monorepo, do some code analysis, split into many repositories. Monorepos are fool's gold unless you can have a team whose sole job is managing monorepo complexity, and even then I might ask "Why waste a team on monorepos?"
The problem is that we have crammed many related but distinct concepts--versioning, package management, access control, issue tracking, project management, licensing, etc.--into a single envelope called the "repo". Having a single top level version and commit-log for all an organization's code is a huge win. But that doesn't mean you necessarily want to manage those other more granular concerns at the top level too.…
I don't really agree. Rather than stuffing all your code into one envelope and then building tooling to decipher it, why not build tooling that allows commit log and versioning information for an arbitrary amount of repositories? Why does code have to live side-by-side to manage metadata about it?
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#93I have a custom script for my monorepo that circumvents sub-modules while getting most of their benefits.
Basically, on git pull it goes through and finds all the folders with `.x_git`, renames them to `.git` and pulls. On push it does the inverse. My submodules are really just copies of one small part of the monorepo, but it allows me to have a monorepo with some parts of it being public.
So far it's been amazing.. there's no real line between sub-repos and sub-folders, if I can push a commit that modifies many with a single PR, it's obvious that monorepo is for internal stuff while the public repo is for public facing discussions... I really like it.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#94Earlier quoted context omitted.
I disagree (unless your "bad environment" gets tautologically redefined to mean any place where people with good intentions don't get good results.... which you might). I was working in a place that employed incredibly smart and capable but highly opinionated people, and did not treat them as cogs, and did let them influence their productivity, happiness, and quality of the code. However, there was a major philosophi…
If "it worked" and "was not a bad environment", I really don't see the issue here (unless your "it worked" gets redefined to mean "it didn't work"... which you might). Sounds like a nice and intellectually stimulating environment, and that they had best intentions. Honestly, someone wanting a dictator has bigger ego problems than those people.
Very long onboarding, impedance mismatches everywhere, tons of unneeded glue logic. Lots of consternation. It worked in the sense that it wasn’t a failure like a significant number of software projects. It could have worked significantly better if the people involved didn’t feel they needed to show off.
And my model for A dictator is Linus Thorvald, Guiro van Rossum, Andreas Rumpf; if you can’t appreciate the management model and abilities that a benevolent dictator brings, we will just have to agree to disagree.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#95I have worked on projects organized both ways with git and my observation is that regardless of the choices, the right tooling can make the workflows a lot more fluent.
For separate repos, share code as much as possible through a packaging system so one does not have to make a lot of refactoring across multiple repos. It sounds backward but an auto minor version update can ease a lot of merging pains in a CI environment.
For monorepos, figure out as early as possible what/how things should be shared and separated. I worked on a project with 20+ services and websites that form a whole product and each service chooses its own languages and build systems but shared deployment interfaces for unified service discovery. CI got tricky as it can be blocked because of an unrelated change. I have not found the unified commit history super helpful as I only have context on a few of the services and most of the time I only look at histories with `git log my/path'.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#96Earlier quoted context omitted.
The problem is that we have crammed many related but distinct concepts--versioning, package management, access control, issue tracking, project management, licensing, etc.--into a single envelope called the "repo". Having a single top level version and commit-log for all an organization's code is a huge win. But that doesn't mean you necessarily want to manage those other more granular concerns at the top level too.…
I don't really agree. Rather than stuffing all your code into one envelope and then building tooling to decipher it, why not build tooling that allows commit log and versioning information for an arbitrary amount of repositories? Why does code have to live side-by-side to manage metadata about it?
That works too. I think you could arrive at the ideal setup from either direction: a monorepo with appropriate boundaries or multi-repos with an “umbrella” versioning layer for all of them. The end result is more or less the same.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#97Earlier quoted context omitted.
If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture. Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your a…
> you need to manage the problems of monorepos If have not seen a problem on monorepos that does not exist on multirepos, other than more quickly reaching tools' repo size limits (but this is becoming less of a problem all the time on e.g. git and mercurial, and supposedly was never a problem with perforce). I have seen many problems multirepos have that monorepos do not - such as that multirepos can (and almost alwa…
> If have not seen a problem on monorepos that does not exist on multirepo
How do you refer to a library in a monorepo from a different repo?
The workflow seems to me to entail an all-or-nothing approach.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#98Earlier quoted context omitted.
> you need to manage the problems of monorepos If have not seen a problem on monorepos that does not exist on multirepos, other than more quickly reaching tools' repo size limits (but this is becoming less of a problem all the time on e.g. git and mercurial, and supposedly was never a problem with perforce). I have seen many problems multirepos have that monorepos do not - such as that multirepos can (and almost alwa…
> If have not seen a problem on monorepos that does not exist on multirepo How do you refer to a library in a monorepo from a different repo? The workflow seems to me to entail an all-or-nothing approach.
I never tried it, but if I needed it, I would easier use sparse submodule checkout (if it’s possible - I don’t know if that’s true), or submodule the entire repo, and the symbolic link into the internal part I need.
If all else fails (e.g. you need 50 files out of a 100GB repo, which makes things too slow), I would set up a git subtree extract as a checkin hook, which would mean there exists another repo/branch automatically updated on checkin - and which you could use any way you want, e.g, simple submodule.
Do note that’s just what I said: the problem, “I only need a small part of the repo” is independent of it being a monorepo or multirepo.
You would only avoid this problem in a multirepo if you cut it to pieces in exactly the right places. it’s just a matter of whether the parts of the repo you don’t need are too large to make this comfortable — more likely in a monorepo as it is larger - but will happen in a big enough multirepo as well.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#99Earlier quoted context omitted.
I don't really agree. Rather than stuffing all your code into one envelope and then building tooling to decipher it, why not build tooling that allows commit log and versioning information for an arbitrary amount of repositories? Why does code have to live side-by-side to manage metadata about it?
That works too. I think you could arrive at the ideal setup from either direction: a monorepo with appropriate boundaries or multi-repos with an “umbrella” versioning layer for all of them. The end result is more or less the same.
I guess I think one of those scenarios comes with all of the benefits but very few of the drawbacks than the other. But you're right, you can skin a cat however you want.
Re: Monorepoize – Bash scripts for creating a monorepo out of smaller repos
#100Earlier quoted context omitted.
And when your CI fails, it blames to a specific commit, rather than "the commit that triggered it, plus any commit in a dependency repo around the same time." And if you need to maintain old release branches, each one is a single branch, and your tooling doesn't need to know anything special about what branches of other repos to check out. And `git bisect` works. All of these things are possible without a monorepo, b…
git submodules tracks the version of the sub repos. So if you organize that your releases are a master repo with all needed repos as sub-modules, your sub-repo version tracking is already done fr you. All that while still allowing sub-repo to move forward faster than other repos. With a mono-repo, you can only allow some sub-system to go forward faster than others by making them live permanently in separate branches.…
Yeah, we have set up a "parent" repo with git submodules for all the individual repos. But it was an afterthought and our workflow has not really caught up to the new possibilities.