Why Google Stores Billions of Lines of Code in a Single Repository (2016) • Hacker News Board

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#231

Post by EnderMB » Wed, Jul 25, 2018, 9:20 AM UTC

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I've seen this a few times in the .NET world, mainly as a carry-over from Subversion when we had moved to Mercurial and git.

Some mad genius in a company will write a fuck-ton of helper classes and utilities that take the heavy lifting out of everything remotely hard, to the point where you almost never need to touch a third-party API for a CMS, email send service, or cloud-hosting provider. Instead of supplying these as private NuGet packages to be installed into an application, they sit in solutions in their entirety, in case they are needed. That application then goes to a new developer team, and they have zero idea why there are millions of lines of code and dozens of projects for a basic website that doesn't really seem to do anything.

It's a nice idea, but it has resulted in some very tightly coupled applications. I remember one time where a new developer changed some code in one of the utilities that handled multi-language support, and for some reason our logs reported that the emails were broke.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#232

Post by lclarkmichalek » Wed, Jul 25, 2018, 9:32 AM UTC

not_kurt_godel wrote: post #193

ebikelaw wrote: post #52

Earlier quoted context omitted.

See, this is where your argument broke down for me. Once you’ve decided there is some library of common code, and assuming you factor out that code into another repo, you’ve just lost your ability to easily make breaking changes to the common code, which is something trivially easy to do in a monorepo. Why would you want that? It seems to me that if you have multiple projects sharing a base of common code then a mono…

> you’ve just lost your ability to easily make breaking changes to the common code It should be hard to make breaking changes in common code. Even 'trivial' breaking changes seem to have a way of breaking things even when they shouldn't. If you need to make a breaking change to common code, the proper way to do it is add the new functionality separately, deprecate the old functionality (i.e. with javadoc so it gets c…

And you can do that in a monorepo. But realistically, there are plenty of trivial breaking changes (renaming Foo to FooX) that don't warrant that effort, and so usually don't get done outside of monorepos.

You should apply care when making breaking changes. Having it be hard is a separate issue - I'd say distractions from multi repo tooling would introduce more risks overall. Having a unified CI system in a monorepo is really nice.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#233

Post by justicezyx » Wed, Jul 25, 2018, 9:43 AM UTC

evfanknitram wrote: post #223

justicezyx wrote: post #220

Earlier quoted context omitted.

Single repo is one design that coherently addresses source control management and dependency management. The key is to let the repo be a single comprehensive source of data for building arbitrary artifacts.

I don't know what this means. How is "single repo" a "design" and how does this design dictate dependency management? Yes, if you have a single repo then that would be a single source of data for building your stuff. That seems redundant.

See Bazel, you have the depes manifested as source controlled the data, then you can build everything as deterministically as possible.

Then you can manage dependency as part of the normal source control process.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#234

Post by thomaslee » Wed, Jul 25, 2018, 9:44 AM UTC

edejong wrote: post #222

jgibson wrote: post #194

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have…

The key here is reverse dependency management. “If I change X, what would influence this change?”. This can be achieved with single repo better than multi-repo due to the completeness of the (dependency) graph.

Exactly this. Or at least it's a way this can be achieved, assuming solid testing & some tooling in the mix.

For folks unfamiliar with it, the issue is something like:

1. You find a bug in a library A.

2. Libraries B, C and D depend on A.

3. B, C and D in turn are used by various applications.

How do you fix a bug in A? Well, "normal" workflow would be something like: fix the bug in A, submit a PR, wait for a CI build, get the PR signed off, merge, wait for another CI build, cut a release of A. Bump versions in B, C and D, submit PRs, get them signed off, CI builds, cut a release of each. Now find all users of B, C and D, submit PRs, get them signed off, CI builds, cut more releases ...

Now imagine the same problem where dependency chains are a lot more than three levels deep. Then throw in a rat's nest of interdependencies so it's not some nice clean tree but some sprawling graph. Hundreds/thousands of repos owned by dozens/hundreds of teams.

See where this is going? A small change can take hours and hours just to make a fix. Remember this pain applies to every change you might need to make in any shared dependency. Bug fixes become a headache. Large-scale refactors are right out. Every project pays for earlier bad decisions. And all this ignores version incompatibilities because folks don't stay on the latest & greatest versions of things. Productivity grinds to a halt.

It's easy to think "oh, well that's just bad engineering", but there's more to it than that I think. It seems like most companies die young/small/simple & existing dependency management tooling doesn't really lend itself well to fast-paced internal change at scale.

So having run into this problem, folks like Google, Twitter, etc. use monorepos to help address some of this. Folks like Netflix stuck it out with the multi-repo thing, but lean on tooling [0] to automate some of the version bumping silliness. I think most companies that hit this problem just give up on sharing any meaningful amount of code & build silos at the organizational/process level. Each approach has its own pros & cons.

Again, it's easy to underestimate the pain when the company is young & able to move quickly. Once upon a time I was on the other side of this argument, arguing against a monorepo -- but now here I am effectively arguing the opposition's point. :)

[0] https://github.com/nebula-plugins/gradle-dependency-lock-plu...

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#235

Post by bananarepdev » Wed, Jul 25, 2018, 9:50 AM UTC

jgibson wrote: post #194

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have…

Maybe this is a reflection of modern tools using the version control system to store built artifacts, like npm and "Go get" do. Anyway, depending on the programming language, you can have a monorepo and still bind your modules with artifact dependecy, not necessarily depending on the code itself.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#236

Post by kungtotte » Wed, Jul 25, 2018, 10:06 AM UTC

kqr wrote: post #186

013a wrote: post #173

Earlier quoted context omitted.

I call this "Google Imposter Syndrome". Because Google (insert Facebook, Apple, Amazon, etc) has success with Monorepos (insert gRPC, Go, Kubernetes, React/Native, etc), it must be a great idea, we should do it. You see this everywhere . Also known as an Appeal to Authority. My personal opinion: very few companies will hit a point where sheer volume of code or code changes makes a monorepo unwieldy. Code volume is a…

Someone at some point said "Google may not be successful for the interview practises they use; they're big enough that they could very well be successful despite the interview practises they use." It stuck with me, and is applicable to so many things. Including, maybe, this?

Another question is just the sheer scale of the FAANG companies, making things work at that scale is likely to be counterintuitive sometimes.

I just looked it up, Facebook has 2.2 billion users monthly. That's almost a third of the entire planet.

Shit that makes sense for them won't make sense for 99% of everyone else.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#237

Post by gberger » Wed, Jul 25, 2018, 10:12 AM UTC

jgibson wrote: post #194

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have…

Yes, Google's internal tool handles permissions based on directory owners.

They use the same OWNERS-file model as in the Chromium project [1], the only difference being the tooling (Chromim is git, google3 is ... its own Perforce-based thing).

[1] https://chromium.googlesource.com/chromium/src/+/lkcr/docs/c...

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#238

Post by nine_k » Wed, Jul 25, 2018, 10:13 AM UTC

threeseed wrote: post #45

nine_k wrote: post #36

Earlier quoted context omitted.

Most folks who consider a monorepo don't have billions of lines of code, and often not even millions. Linux kernel is a monorepo.

Linux kernel is one functional piece of work though. Imagine if we combined KDE, Gnome, Linux Kernel, ZFS etc all in the one monorepo.

Where there are obvious and pronounced functional boundaries, often backed by administrative boundaries, separate repos makes total sense.

Otherwise, it's an optimization; see "premature optimization" for cautions.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#239

Post by justicezyx » Wed, Jul 25, 2018, 10:15 AM UTC

MaxBarraclough wrote: post #228

justicezyx wrote: post #220

Earlier quoted context omitted.

Single repo is one design that coherently addresses source control management and dependency management. The key is to let the repo be a single comprehensive source of data for building arbitrary artifacts.

> The key is to let the repo be a single comprehensive source of data for building arbitrary artifacts. By that do you mean it's one way of doing it, or that it's the only way? Seems clear to me that it's not the only way. For instance .Net code tends to be Git for the project source + NuGet for external dependencies. It works pretty well.

It's one way. There isnt any problem that can only be solved in one way.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#240

Post by yehosef » Wed, Jul 25, 2018, 10:29 AM UTC

tiglionabbit wrote: post #83

acomjean wrote: post #41

Earlier quoted context omitted.

But what are the alternatives to the monorepo in git? All the ways of splitting code up and deploying multiple git repos for one project seem terrible.

Fun fact. I asked Facebook why they built their monorepo on Mercurial instead of Git. They said there were scaling issues in Git that made it unusable for large repos and the Git maintainers would not work with them to fix these issues. However, they were able to work with Mercurial to make it capable of holding their entire company in one repo.

Here's the article (or one of them) https://code.fb.com/core-data/scaling-mercurial-at-facebook/