Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

171–180 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#171

Im sure properly organized it's okay, but from what I've seen it's mediocre at best, especially with legacy/technical debt it's a huge mistake. Start breaking that repo apart, because it probably isn't very/hopefully depending on the debt that exists.

One of the big advantages of the monorepo is actually that it prevents technical debt from accumulating. If a change somewhere else breaks your code, you can't put off dealing with it -- you are forced to fix the issue immediately.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#172

Earlier quoted context omitted.

Same line of thinking, just different conclusions. I feel terrible for anyone trying to run a company with open-source style independent repos. On a popular github project, you have MANY potential contributors that will tell you if a PR, or a release candidate break API compatibility, etc. There are thousands of hours in open source dedicated to fixing integration issues due to the (unavoidable) poly-repo situation.…

working as dev with academic teams, I usually use many repos for "damage control" as git-ignorant scientists will dump irrelevant files into a repo. with that in mind, is monorepo is a universally good approach or is more dependent on good behavior of team members than polyrepo?

I don't think that there is the one size fits all solution especially if you can't expect basic knowledge about git

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#173
post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I call this "Google Imposter Syndrome". Because Google (insert Facebook, Apple, Amazon, etc) has success with Monorepos (insert gRPC, Go, Kubernetes, React/Native, etc), it must be a great idea, we should do it. You see this everywhere. Also known as an Appeal to Authority.

My personal opinion: very few companies will hit a point where sheer volume of code or code changes makes a monorepo unwieldy. Code volume is a Google-problem. But every company will have problems with Github/Gitlab/whatever tooling with multiple repos; coordinating merges/deploys across multiple projects, managing issues, context switches between them, etc. And every company will also have problems with CI/CD in a monorepo.

Point being... there are problems with both, and there are benefits to both. I don't think one is right or wrong. I personally feel that solving the problems inherent to monorepos, at average scale, is easier than solving the problems inherent to distributed repos. The monorepo problems are generally internal technical, whereas the distributed repo problems are generally people-related and tooling outside of your control.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#174
Sorry, but as someone who has been in orgs that do both, mono repo is a mistake. Constant needs to pull unrelated changes before pushing, pipelines requiring to grab the whole repo for dependencies, etc. I understand the arguments for mono repo, but never think it's nothing that outweighs the cons.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#175
It's not just devops that you need to pull off a large monorepo; the other big thing is a strong testing culture. You have to be able to rely on unit tests from across the code base being a sufficient indicator of whether your commit is good. AND a presubmit process that can compute which parts of the monorepo get affected by your diff, and run tests against them automatically before committing your diff.

Google not only has the above but also has a strong pre-submission code review process which catches large classes of bugs in advance.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#176
post #58

Earlier quoted context omitted.

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

What you're proposing goes a step beyond multiple repos and into package versioning. That is one viable workflow: Make a change to the common code and publish it as a new package version while allowing all existing code to continue to use the old package. Then, migrate other projects to the newer version of the dependency one by one. Allowing multiple versions of the same code to exist in production at once adds comp…

It almost certainly depends on the company. Consider General Electic's microwave oven firmware and hydroelectric control station software. Both might actually share some code. Maybe they both use GEEventLoop or GEHardwareButton or something. But there's no reason to be concerned about having different versions in production at once.

I don't think there's a universal answer to your question.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#177
post #173
post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I call this "Google Imposter Syndrome". Because Google (insert Facebook, Apple, Amazon, etc) has success with Monorepos (insert gRPC, Go, Kubernetes, React/Native, etc), it must be a great idea, we should do it. You see this everywhere . Also known as an Appeal to Authority. My personal opinion: very few companies will hit a point where sheer volume of code or code changes makes a monorepo unwieldy. Code volume is a…

I've seen multiple companies struggling with maintaining interdependencies between multiple repos. It often results in an expensive custom solution. As a general guideline I'd say "when in doubt, put the code in a single repo"

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#178
post #104

Earlier quoted context omitted.

Incrementally monolithic CL number is also useful. You can mark quite a lot of things with it - not only binary releases, but other developments too (configuration files, etc.). At the end your binary "version" comprises of main base CL + cherrypicked individual CL's - rather than branch with these fixes - I guess one can encode this too with git/hg - by using sha hashes - but this becomes much bigger in terms of inf…

Most popular Git frontends (GitHub and GitLab too, I believe) let you link to commits with just the first 5-6 characters of the hash. I don't think that's much different to remember than a Perfore CL number.

To me the issue is when mentally trying to work with these numbers, P4 & G4's numbers increment, so I can tell which one came before the other - I can't do this with hashes. I'm sure I can get used to the other way, but this cannot easily be ignored.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#179
post #98
post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Spot on! I've seen org wide mono repos at Microsoft and they had their custom tooling and build systems built on top of SourceDepot.

Which is just rebadged Perforce :)

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#180

Earlier quoted context omitted.

> multiple git repos for one project If it's one project, it's not a monorepo. It's a repo.

We wanted to have a "common" subsystem that was common across projects. Being able to add and work on the common area and new projects at the same time was important. Pushing the common area back and being able to deploy to the older projects and test was important. This seems difficult in git. There are "submodules" and "subtrees" but none seemed particularly great and as far as I could tell each came with a bunch o…

[deleted]
Post reply on HN