[deleted]
We Put Half a Million Files in One Git Repository, Here’s What We Learned
231–240 of 270 posts
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#232Earlier quoted context omitted.
You can put microservices in a monorepo. You just put them in their own folder.
Of course, but can you split a monolith across multiple repos?
Sure. It'd drastically complicate the build process and CI for little benefit compared to other approaches (e.g. sparse checkout), but you could even in principle create a repo for each code directory and stitch them all together.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#233I get a headache imagining everything Google in a monorepo. So how do you reconcile this single GIT repository approach with Infoworld's "The case against monorepos"? [1]: Reason #1. Monorepos go against single-team ownership principles Reason #2. Monorepos encourage bad practices involving massive refactoring Reason #3: Small repositories are better than large ones [1] https://www.infoworld.com/article/3638860/the-c…
I think the biggest problem here is that git (out of the box) is not well suited to a monorepo.
But in terms of the reasons in "The case against monorepos":
> Reason #1. Monorepos go against single-team ownership principles
I think it's up for debate about whether or not single-team ownership is desirable. But even if it is, I don't see the difference. Just have teams own their directories within the monorepo.
If you want to enforce that a team owns a particular part of the repo, just put some rules into the code review tool to ensure that a change to that component can't be merged without someone from that team reviewing/approving it.
> Reason #2. Monorepos encourage bad practices involving massive refactoring
Again, this just appears to be the author's opinion that massive refactoring is somehow problematic, without providing much of an argument against it. If you're changing service APIs, then sure, you need to be careful about the order in which you roll out changes. But if you're changing the APIs of shared libraries, then being able to do a single large refactoring is absolutely valuable.
> Reason #3: Small repositories are better than large ones
This is just a tooling issue. You can still separate projects by directories, and if your tools allow you to just check out a subdirectory, or if you have some sort of virtual filesystem on top of your source control, then you don't have to pay the penalty of pulling down the entire repository.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#234I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expens…
Yeah, I think choosing git is fine, and choosing to have a monorepo is fine, but you probably don't want to do both.
Or, at the very least, once your git monorepo reaches a certain size, you should either split it up, or switch to something that handles monorepos better. Even if that something is e.g. a virtual filesystem on top of git.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#235Earlier quoted context omitted.
How do you split your XLIFF files? Does each project get one big one and the proliferation is simply due to number of languages, or do you have a more granular split (eg. if you've got one component, it will have dozens of XLIFF files for every language, instead of one per language)? By the numbers you mention, 70% of the files make a ratio of code files to translation files 1-3, so unless you only support 3-5 langua…
It's one XLIFF file per locale per component, not including the source en_US. We currently support 104 locales. More info: https://news.ycombinator.com/item?id=28931601
Thanks: that's still a lot of components (3000+) if you've got at least 300k xlf files!
Good job managing all that regardless of the approach :)
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#236Earlier quoted context omitted.
They have obviously invested a lot over time into streamlining their build process: so much so that they're putting an article about it. All of the problems they are having are basically due to their use of a monorepo: they do explain that they made the decision early, but I wonder what are the advantages over multiple repos they are seeing that it was worth it all this trouble?
We can talk about the advantages of monorepos, but your questions is phrased in a way that makes me think that you don't see any "trouble" in multiple repositories. I would encourage you to do some research and keep an open mind.
I see "trouble" in all approaches: their solutions to manage their git monorepo approach it by switching to multi-repo emulation.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#237Earlier quoted context omitted.
They have obviously invested a lot over time into streamlining their build process: so much so that they're putting an article about it. All of the problems they are having are basically due to their use of a monorepo: they do explain that they made the decision early, but I wonder what are the advantages over multiple repos they are seeing that it was worth it all this trouble?
They seem to have purposely left that out of the scope of this article. There are myriad articles about the benefits and downsides of monorepos vs multiple repos.
In another sibling comment I mentioned how I looked through their engineering blog and didn't see any post where they have talked about any of the benefits they are enjoying due to their use of monorepo.
The question is specifically about their usecase, since not everybody would hit the same bottlenecks as they did with git monorepos.
Iow, have they stopped and thought whether it's still worth it (eg how often do their engineers make use of the monorepo benefits like cross-project refactorings)?
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#238It took me a while to find out what Canva actually does, but from https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure,…
I think it is a crowd dumbing effect. Since hundreds of engineers sharing the mono-repo, no one can or care to make the decision or is able to push the decision for alternate. Even when every one is complaining, it is still far from every one agreeing on the alternate. Crowd settle at the lowest denominator.
Re: We Put Half a Million Files in One Git Repository, Here’s What We Learned
#239This reminded me of https://github.com/twosigma/git-meta
Yeah, git-meta is a reasonable alternative to megamonorepos. It's basically a repo-mega-forest.
A few things:
- your build system will need more code
to deal with a repo-mega-forest than
a megamonorepo
- code indexers may not be able to see
cross-repo dependencies
You'll have pain no matter what. My preference would be for a megamonorepo approach to scale properly. That means partial/sparse cloning, as well as shallow cloning, and also all the hacks that are supposed to make git-status and git-log (and git-blame, and...) fast in partial clones.