Because Google does something, doesn't mean it's a good thing to do for anyone else. This kind of infrastructure is very expensive to maintain, and suffers from many flaws like -almost- everyone being stuck using SDKs that are several versions behind the latest production one even for the internal GCP ones.
Google stores billions of lines of code in a single repository (2016) [pdf]
31–40 of 209 posts
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#32Earlier quoted context omitted.
If you had every line of code that Google wrote, what would you do with it? But I found this discussion on HN. https://news.ycombinator.com/item?id=11790438
I imagine looking for vulnerable areas of the code might be something people would be interested in doing. Maybe start with login or billing or something. You could also look at recent activity to spot new, unannounced projects. You could use blame to find who wrote what and target them for anything from job offers to social engineering attacks.
Most of that information is readily available on the corporate intranet without having to dig through source code.
Security-by-obscurity isn't something to rely on (again, except in the case of things like abuse detection where there's no alternative).
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#33Earlier quoted context omitted.
They tried that by making a bit available with a remote cloud builder for Bazel. It failed for some reason and they pulled it. I think building something that scales for one big repo is just a completely different problem than making it scale for a lot of small repos.
Bazel is not failing in the open source world though
I think that maintaining a hosted service has significantly higher fixed costs than maintaining an open source project whose users are responsible for deploying it themselves. So a higher degree of adoption would be necessary to justify it.
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#34Monorepos are great... but only if you can invest in the tooling scale to handle them, and most companies can't invest in that like Google can. Hyrum Wright class tooling experts don't grow on trees. A good article to reference when this topic gets raised: http://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-yo...
You don't need google scale tooling to work with a mono repo until you are actually at google scale. Gluing together a bunch of separate repos isn't exactly free either. See, for example, the complicated disaster Amazon has with brazil. In the limit, there are only two options: 1. All code lives one repo 2. Every function/class/entity lives in its own repo with a third state in between 3. You accept code duplication…
This mostly seems like a problem for pure library code. If some bit of logic is only needed by a single independently-released service, then there's no reason not to put it in that service's repo.
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#35There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern.
1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between services, which is absolutely necessary for the stability of mission-critical business services.
2. It encourages poor API contracts because it lets anyone import any code in any service arbitrarily. Shared functionality should be exposed as a standalone library with a clear, well-defined interface boundary. There are entire packaging ecosystems like npmjs and pypi for exactly this purpose.
3. It encourages a ton of code churn with very low signal. I see at least one PR every week to code owned by my team that changes some trivial configuration, library call, or build directive, simply because some shared config or code changed in another part of the repo and now the entire repo needs to be migrated in lockstep for things to compile.
I've read this paper, as well as watched the talk on this topic, and am absolutely stunned that these problems are not magnified by 100x at Google scale. Perhaps it's simply organizational inertia that prevents them from trying a more reasonable solution.
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#36[flagged]
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#37I’d really love to know what the breakdown of those 2 billion lines of code is by product. What a huge number.
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#38> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world. > Access to the whole codebase encourages extensive code sharing and reuse [...] Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.
Despite almost everything being in one big repo, it has silos. Not everyone has read access to everything. Some code, like the important bits of Search, is only available on a need-to-know basis.
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#39Monorepos are great... but only if you can invest in the tooling scale to handle them, and most companies can't invest in that like Google can. Hyrum Wright class tooling experts don't grow on trees. A good article to reference when this topic gets raised: http://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-yo...
i’ve made this argument before, but you can run a 1k engineering company in a monorepo with the tools and services that exist today. between improvements to bazel (and alternatives) and adjacent tooling like build caching/target diffs, core git scalability, merge queues, and other services you can just plug things together over a few days/as needed and it will just work.
all of the stuff that you can’t do easily yet (vfs for repo, remote builds) just isn’t relevant enough at this scale.
Re: Google stores billions of lines of code in a single repository (2016) [pdf]
#40Monorepos are great... but only if you can invest in the tooling scale to handle them, and most companies can't invest in that like Google can. Hyrum Wright class tooling experts don't grow on trees. A good article to reference when this topic gets raised: http://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-yo...
You don't need google scale tooling to work with a mono repo until you are actually at google scale. Gluing together a bunch of separate repos isn't exactly free either. See, for example, the complicated disaster Amazon has with brazil. In the limit, there are only two options: 1. All code lives one repo 2. Every function/class/entity lives in its own repo with a third state in between 3. You accept code duplication…
> You don't need google scale tooling to work with a mono repo until you are actually at google scale.
I really don't see how that would work for most companies in practice. Most of the off the shelf tooling used by companies with hundreds or thousands of developers assumes working with polyrepos. It's good we're seeing simpler alternative to Bazel but that's just one piece of the puzzle.