71–80 of 166 posts
Earlier quoted context omitted.
the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...
yeah by writing custom version control software. Am I going to convince my company to do that (which has like 50k software engineers) probably not.
Earlier quoted context omitted.
It's not, why do I have to checkout terabyte of code that I don't need, even if the code is modularized?
No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.
Again: I wish it was a smooth experience. Because I like the ideas very much. But it wasn't when I tried and I don't know anyone -- outside of Google -- for whom it was a smooth experience.
There aren't good monorepo solutions out there (yet). Git LFS is great for few large files, but it doesn't help with tons of smaller files. Git submodules are crap when it comes to usability, and have been for a long time, it's even mentioned in the famous Torvalds Git Talk. Git had a sparse checkouts feature since a long time, but it only affected the checkout itself, all the blobs would still be synced. Now, Git is…
Earlier quoted context omitted.
This has nothing to do with monorepos though. Its entirely a company policy issue. There's nothing about the monorepo that prevents you from writing a script that ran on precommit and built and deployed via docker to a test cluster. Unless you mean your presubmit test would push to production machines, that's bad and shouldn't be allowed, but again has nothing to do with a monorepo. A company could just as easily hav…
If a tool begets using it wrong all the time, then after a certain point, it’s the tool’s fault. What you’re saying amounts to something of a No True Scotsman fallacy... “no _real_ monorepo would limit different projects from using individualized tooling if needed...” Yet that limitation suspiciously coexists with monorepo tooling frequently, and does not frequently coexist with polyrepo tooling.
This is the (wrong) assumption. Like I said, there's nothing about a monorepo that "begets" draconian policy. Your anecdotal experience is not a rule. The monorepo I work with doesn't have draconian policies about how tooling must work. There are apis and recommended tools, and if those don't fit your needs (which is unusual), the teams that maintain those tools are willing to support your uses, but if not, you're also free to hack yourself something that works. Writing additional pre-commit hooks is encouraged.
> What you’re saying amounts to something of a No True Scotsman fallacy
Again, no. Certainly monorepos can do this. They're still real monorepos. But polyrepos can too. They're still polyrepos. Its orthogonal.
"Shared responsbility" is one of those ideas that sounds good on paper, but doesn't really scale terribly well in the real world. As the old saying goes "when everybody is responsible, nobody is responsible". More to the point, as the author of TFA allows, once a system reaches a certain size, nobody can understand it all. At some point you have to engage division of labor /specialization, and once you do that, it do…
Here's my rule: You break it, you fix it.
> I'd rather see a poly-repo approach, with a designated owner for discrete modules, but where anybody can clone any repo, make a proposed fix, and submit a PR.
I'd rather see pairing, extensive tests and fast CI. I see PRs as a necessary evil, rather than a good thing in themselves. If I make a change that breaks other teams, I should fix it. If I can make a change to fix code anywhere in the codebase, I should write the test, write the fix and submit it.
Small, frequent commits with extensive testing creates a virtuous cycle. You pull frequently because there are small commits. You are less likely to get out of sync because of frequent pulls. You make small commits frequently because you want to avoid conflicts. Everyone moves a lot faster. I have had this exact experience and it is frankly glorious.
Earlier quoted context omitted.
No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.
I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with e…
I find Bazel’s syntax much easier to deal with than other build languages that use JSON (essentially the same Python syntax but with lots of extra quotes everywhere and extra fussiness about where commas are allowed).
Earlier quoted context omitted.
If a tool begets using it wrong all the time, then after a certain point, it’s the tool’s fault. What you’re saying amounts to something of a No True Scotsman fallacy... “no _real_ monorepo would limit different projects from using individualized tooling if needed...” Yet that limitation suspiciously coexists with monorepo tooling frequently, and does not frequently coexist with polyrepo tooling.
>If a tool begets using it wrong all the time This is the (wrong) assumption. Like I said, there's nothing about a monorepo that "begets" draconian policy. Your anecdotal experience is not a rule. The monorepo I work with doesn't have draconian policies about how tooling must work. There are apis and recommended tools, and if those don't fit your needs (which is unusual), the teams that maintain those tools are willi…
I’d flip it around and say instead that you are assuming the properties by which to compare the two approaches ought to be properties that are roughly like “first principles” and that no first principles difference really exists between them in terms of limiting what you can do.
But this is the wrong way to look at it because, pragmatically, it’s simply just not the sociological phenomenon that actually happens as a side effect in terms of the practical result. Who cares if there’s a first principles reason for them to be different in terms of effectiveness? I certainly don’t— they just are different in terms of effectiveness.
There aren't good monorepo solutions out there (yet). Git LFS is great for few large files, but it doesn't help with tons of smaller files. Git submodules are crap when it comes to usability, and have been for a long time, it's even mentioned in the famous Torvalds Git Talk. Git had a sparse checkouts feature since a long time, but it only affected the checkout itself, all the blobs would still be synced. Now, Git is…
From earlier discussions around monorepos, I saw references that Google, Facebook, and other large monorepo orgs have been making use of Mercurial.
I think that the reason of the argument is that both authors understand the definition of "large repository" very differently. With 100 engineers a monorepo might seem a good idea. With 500 it becomes nearly impossible to do anything involving a build. Some isolation is needed. Also from my experience many engineers just don't give a shit about architecture. They create entangled mess, that kind of works for the cust…
the fact that essentially 100% of big tech companies use monorepos seems like evidence that it is at least possible to do it in a scalable way...
Today, not quite. I work for a multi billion dollar tech company and we have several thousand repos (and it's awesome)
Earlier quoted context omitted.
Basically because for certain projects and teams, the effort to package internal / proprietary libraries and other similar dependencies can be much larger than the benefit. Packaging is effort. You decide to cut a release, stamp a version number, write a changelog, package and distribute it, and then backport fixes into a long-running branch. This effort makes a lot of sense if your consumers are complete strangers w…
I really don't understand what you describe as effort or huge burden. Writing a simple script that can solve your releasing tasks is simple. Imo a lot of engineers just want to write code but a lot of the time building software consists of other things too, such as testing, releasing, documentation etc. Simply avoiding them doesn't make it better.
I’m also completely baffled by your statement that “simply avoiding them doesn’t make it better.” Reading that statement, I can only feel that I have somehow failed to communicate something and I’m not really sure what, because it seems obvious to me why the premise of this statement is wrong. When you avoid performing a certain task, like releasing software, which costs some number of work hours, you can reallocate those work hours to other tasks. It’s not like the tasks of releasing and versioning simply stop happening, but you also get additional hours to accomplish other things which may be more valuable. So it’s never an issue of “simply avoiding” some task, at least on functional teams, the issue is choosing between alternatives.
And it should also be obvious that cutting discrete releases for internal dependencies is not an absolute requirement, but a choice that individual organizations make depending on how they see the tradeoffs or their particular culture.
There really are many different ways to develop software, and I’ve seen plenty of engineers get hired and completely fail to adapt to some methodology or culture that they’re not used to. The polyrepo approach with discrete releases cut with version numbers and changelogs is a very high visibility way of developing software and it works very well in the open source world, but for very good reasons many software companies choose not to adopt these practices internally. It’s very sad when I see otherwise talented engineers leave the team for reasons like this.