Let me give a concrete example. The Android open source project (AOSP) which builds the system of Android devices has the code size close to the scale of tens of GB (let alone all the histories!). It is already a massive monorepo in itself. And typically you would have many of them from different OEM/SoC vendors of different major releases. In such a scenario, it would turn into 'a monorepo of monorepos,' which is quite unpleasant to imagine.
Monorepo: please do
91–100 of 166 posts
Re: Monorepo: please do
#92Earlier quoted context omitted.
Sure but then you only have some small portion of the total infrastructure, which adds its own layer of complexity for the people reviewing your changes :P It's all trade offs, is all I'm saying - I honestly still can't decide between the two, although for all companies sub 20 people, I'd for sure stick with a single repo.
If I'm working on Application X, wtf do I care about infrastructure code? Or for that matter, as a specific... if someone is working on Google Maps, should they care about the codebase for Google Inbox for Android?
So - what's stopping you from depending (using) anything else? Or how to stop you from doing this? BAZEL (blaze) has visiblity rules, which by default are private - e.g. the rules in your packages are hidden, unless explicitly made public, or alternatively you can white-list one by one which other packages (//java/com/google/blah/myapp) can include you back.
Let's say there is a new cool service, and your team wants to try it out... but it's not out there for everyone to use, it's in alpha, beta, whatever stage. So you ask for permission from the team, or simply create a CL with your package target, name, "..." folder resolution so that you are whitelisted - eventually you will (if that's good idea, and approved). For example you want, if some library got deprecated, and has been slowly replaced with another, and then now instead of being "//visibility:public" is just white listing the last users of it... Well probably not good idea to be added on that list, as the whole thing is going out soon (yes, Google tends to deprecate internally even faster than externally - ... which is good!). But such mechanisms are helpful in getting this worked correctly.
Re: Monorepo: please do
#93I have personally migrated a medium size polyrepo code base (something like ~20 repos?) into a monorepo and I agonized over the decision. But it lifted a huge weight off my shoulders. I feel like if you are working completely in the open-source world, and you are contributing one open-source project to a larger array of available projects, then the decision to use a polyrepo makes a lot of sense. You can submit libra…
Re: Monorepo: please do
#94Earlier quoted context omitted.
> Fighting back against monorepo design is dangerous What's dangerous about it? Monorepos have a lot of benefits, and should absolutely be considered. Maybe even by most. But right now in the community it's almost pushed as the "only true way with all benefits and no drawbacks", and that's absolutely not true. To the point the knowledge of why and how to poly repo is already starting to get lost. That's dangerous.
The real danger here is anyone talking about any system architectures or tooling as "dangerous" (or "not dangerous") absent any other context... What do you even mean by "dangerous"? To a business? To your health? What is the deal with people trying to make these sorts of global assertions in a vacuum about what's "good" and "bad"? This doesn't make any engineering sense in any way to me. You have a problem and you f…
If the "no one-size-fits-all" claim happens to be genuinely and axiomatically true for a particular engineering trade-off, then fine. There's no one correct displacement of an internal combustion engine. There's no one correct resolution of an LCD screen. Fine. It's demonstrably true that a trade space exists.
But a lot of times people seem to just throw up their hands and call it a trade space when really they just haven't reached a conclusion yet. "There's nothing inherently better or worse between Ubuntu and Windows, they're basically just ice cream flavors!" No! Maybe we haven't fully realized a more perfect operating system yet to settle the debate, but that doesn't just make it a meaningless question. It's perfectly possible for a system to be architected poorly given both the real world it has to interact in and the future world it makes possible. To say that this question is an unanswerable matter of taste is to be completely unimaginative about how good an operating system _COULD_ be. (See the death of operating system research and all that).
CVS is _worse_ than git. It just is. I don't want to hear this "well maybe if it fits your use case" mumbo jumbo. If you think that you have a unique snowflake reason that CVS is more appropriate than git, than you are almost certainly lying to yourself or misinformed.
And it's strict hierarchies like that that inspire these articles. There are a lot of technologies out there, and lot of ideas, and most people don't know most of the things you need to know to come up with a good answer to what suits "their specific situation". So people like myself are looking for lessons learned and certain invariants that help them narrow the solution space. I have no idea whether a monorepo would work well for my organization, and if the only thing that your article has to contribute is "monorepos sometimes work for some people, but YMMV! Good luck!" then I have learned nothing. But if somebody thinks that they've learned a fundamental truth about the universe, that that could be useful to me. Whats more, most people like me have a situation that _isn't_ that specific. We have to write some code, there's some ML shit in there, and some real-time critical stuff in there. Nothing mindblowing. _Most_ software shops shouldn't need something that is particularly bespoke. So coming in with the prior that everybody will have to do something unique to their organization is bizarre. There is so much commonality between what each software company does, in fact, that if a commonly used technology can be used by shop A but legitimately can't be used by shop B, there's a decent chance that this is a problem or limitation with the tech.
So who knows, maybe saying monorepos are _always_ better or _always_ worse really is too ambitious. But I don't think the concept that they _could_ be is a priori ridiculous. End this software relativism! Things can be made better! Yes, strictly better!
Re: Monorepo: please do
#95You can certainly achieve this with Perforce, SVN, HG, any repo system there too.
Linux: FUSE + ?
Windows: Dokan? CBFS? Or the new fangled https://docs.microsoft.com/en-us/windows/desktop/projfs/proj... which VFSForGit uses
Re: Monorepo: please do
#96I have personally migrated a medium size polyrepo code base (something like ~20 repos?) into a monorepo and I agonized over the decision. But it lifted a huge weight off my shoulders. I feel like if you are working completely in the open-source world, and you are contributing one open-source project to a larger array of available projects, then the decision to use a polyrepo makes a lot of sense. You can submit libra…
BAZEL has WORKSPACE file that can work with multi-repos, but AFAIK things are still rough there, though would get better eventually (I'm bit hand-wavy on the details).
If your project is mostly something like C++ (which has support built-in to Bazel) then the WORKSPACE rules will be much more manageable and partial checkouts become a lot easier.
Re: Monorepo: please do
#97Earlier quoted context omitted.
No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.
I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with e…
The Bazel rules for languages is also not perfect imo. Like I dislike hooking Bazel up to tools like NPM and Webpack. I'd rather have a system that could sync NPM modules into third_party automatically and setup Bazel files for them, then have a bundling system that is native to Bazel that allows taking full advantage of it's caching and pure building.
Bazel is imperfect on Windows as well. I have tried to help but admittedly it is hard work and it'll take time. I wanted to get Bazel Watcher working on Windows, but my PR is stalled because the Windows API is very truly quite maddening at times. (Feel free to find the PR, it's almost hilarious how convoluted it is to effectively kill a tree of processes. Linux of course is imperfect here but it lets you get 95% of the way Much easier.)
However, here's what I will say: if you are in an organization, I think Bazel really shines. If you can take time to write some custom tools and rules and really integrate your software into Bazel, it can be an awesome experience. Sadly the publicly available rules try pretty hard to match existing semantics and fall short of showing off how nice Bazel can be in some cases, but I think C and C++ is a great area where Bazel shines above the pack.
Another plus: it is Amazing having a build system that crosses languages. Does your Python script depend on a C module and connect over TCP to a Go program? No problem, all of that is easy to express. Do you want to have a Go script that writes a TypeScript file that gets compiled and bundled into your apps JS bundle? Once again this is all fairly natural and you can easily accomplish it with a simple combination of normal build rules and a genrule.
And Starlark is a reasonably complete almost-subset of Python, so it's easy to compose, extend and refactor your rules. If you want to generate a matrix of targets for say, testing across browsers and platforms, you can do that, and make it reusable too.
Basically my advice with Bazel:
- Check out how well it works with C and C++, and I think Java also works quite well. This should give you an idea of how it looks when done right.
- Don't constrain yourself to what Bazel offers in terms of rules. Starlark is hugely powerful and you can easily make your own rules for things.
P.S.: the weird path syntax is probably many parts legacy, but it's not actually super hard to understand. When you see a colon, the left side of the colon is a path to a folder, and the right side is a target name. When you see double slashes, it means absolute path relative to root of workspace. If the colon is omitted the target name is assumed to be the same as the folder name.
//:base -> the base target in the BUILD file in the root of the workspace
//base -> //base:base -> the base target in the BUILD file in the base folder relative to the of the workspace
//app/ui:tests -> the tests target in the BUILD file in the app/ui folder relative to the workspace root
:genfile -> the genfile target in the BUILD file in the current directory
There is some context sensitivity about how to refer to files versus targets and whether you're referring to runfiles, output files, or build files, but most of the time it's surprisingly obvious actually. When it comes to files versus targets, it largely works a bit like Make except there's namespacing for input files vs output files (and runfiles, but that's another topic.)
There is also an @ syntax used to refer to paths outside the current workspace. It mainly comes into play when importing rules.
Re: Monorepo: please do
#98Earlier quoted context omitted.
> With 500 it becomes nearly impossible to do anything involving a build. Both FB and Google have more than 500 devs and are using a monorepo.
At what cost? Both FB and Google employ hundreds of devs to work on internal tooling only. For most companies this isn't feasible.
Re: Monorepo: please do
#99Earlier quoted context omitted.
> except that separating the PRs makes each isolated set of changes more atomic and focused It makes it less atomic if you need simultaneous changes in multiple repositories. > Had we been able to set this up as a separate repo where there were no global rules over how all compute jobs must be organized, and used our own choice of deployment (containers) with no concern over whatever other projects were using / doing…
> It makes it less atomic if you need simultaneous changes in multiple repositories. No, each individual set of changes is more atomic (smaller in scope, mutating a system from one state of functionality to a new state of functionality). The problem is that it’s a linguistic fallacy to act like in the monorepo case “the system” is the sum of a bunch of separate systems (it isn’t, because they are not logically requir…
I don't see what's superficial about "everything everywhere is in sync", myself.
And I have absolutely seen PR race conditions. Assuming that everyone perfectly sliced up the polyrepo on the first go is optimistic.
Re: Monorepo: please do
#100Earlier quoted context omitted.
But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...
I don't understand. Where is the argument for more productivity?
We keep pruning and gc'ing with different flags, but pulls just seem far slower than other smaller repos.