Why Google Stores Billions of Lines of Code in a Single Repository (2016) • Hacker News Board

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#101

Post by malkia » Wed, Jul 25, 2018, 2:22 AM UTC

testcross wrote: post #100

I don't understand why gitlab/github/bitbucket don't provide better tools for monorepo. This is a topic pretty trendy. But there is absolutely no tools helping with control access, good ci, ...

What's missing in these is cross-reference, which is not possible without somewhat established BUILD system (caps "pun-intened") - e.g. like bazel/build, then a source code indexer, etc, etc.

This becomes very critical for doing reviews, since it allows you to "trace" things without running them, apart from many other things. For example large scale refactorings looking for usages of functions, and other examples like it.

Why githab/gitlab/etc. can't do it? Well because hardly there could be one encompassing BUILD system to generate correctly this index.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#102

Post by acomjean » Wed, Jul 25, 2018, 2:24 AM UTC

reificator wrote: post #46

acomjean wrote: post #41

Earlier quoted context omitted.

But what are the alternatives to the monorepo in git? All the ways of splitting code up and deploying multiple git repos for one project seem terrible.

> multiple git repos for one project If it's one project, it's not a monorepo. It's a repo.

We wanted to have a "common" subsystem that was common across projects. Being able to add and work on the common area and new projects at the same time was important. Pushing the common area back and being able to deploy to the older projects and test was important.

This seems difficult in git.

There are "submodules" and "subtrees" but none seemed particularly great and as far as I could tell each came with a bunch of caveats.

I'll admit my Git skills aren't great, but I've used a variety of source control and tried to suss out the best way to deal with a small team.

We ended up using "git subrepo" which is an add on thing I don't love, but it works.

part of the motivation is "common" and "project 2" are to be open sourced, but "project 1" which also uses "common" isn't.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#103

Post by joshuamorton » Wed, Jul 25, 2018, 2:25 AM UTC

flukus wrote: post #96

robert-brown wrote: post #71

Earlier quoted context omitted.

Yes, that's what you do. And you commit the change when all the tests pass.

And that's insane, people don't scale like that. It's harder enough keeping your head around one large project let alone every project a company has that you might have to jump into at any point.

What exactly makes automated refactoring difficult?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#104

Post by malkia » Wed, Jul 25, 2018, 2:26 AM UTC

a-dub wrote: post #55

It should be noted that the monolithic model is somewhat encouraged by the client mapping system in Perforce, which was Google's first version control system so it is unclear to me if this was deliberate or just a side effect of the best VCS of the time. I also still have doubts around the value of a monorepo, in the article they claim it's valuable because you get: Unified versioning, one source of truth; Extensive…

Incrementally monolithic CL number is also useful. You can mark quite a lot of things with it - not only binary releases, but other developments too (configuration files, etc.). At the end your binary "version" comprises of main base CL + cherrypicked individual CL's - rather than branch with these fixes - I guess one can encode this too with git/hg - by using sha hashes - but this becomes much bigger in terms of information, and human handling it.

I guess not very strong point, but using CL numbers (I'm working with perforce mostly these days) makes things easier. And having one CL monothonically increasing all over all source code you have even better - you can even reference things easier - just type cl/123456 - and your browser can turn it into a link. Among many other not so obious benefits...

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#105

Post by Eridrus » Wed, Jul 25, 2018, 2:29 AM UTC

whack wrote: post #87

anyfoo wrote: post #59

Earlier quoted context omitted.

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" the…

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, al…

> The next morning, all your services end up broken as a result.

I mean, this is the argument for having good integration tests.

At some point someone has to figure out if the new code will break a system; if you don't have good integration tests you're basically left eyeballing the changes, and sure eyeballing changes can work fine in small teams, but at some point you need good tests.

I did some work on a ranking system last year, and by definition there's no way to roll that out incrementally, because, well, it is the central component deciding what thing to do/show, and you have to change the world at once, there is literally no other option. So you need good ways of evaluating these wide reaching changes.

"If you liked it then you shoulda put a test on it" :)

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#106

Post by joshuamorton » Wed, Jul 25, 2018, 2:29 AM UTC

flukus wrote: post #80

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

> and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo That's a huge understatement. They haven't just slapped a few scripts on top of git/svn, they've created their own proprietary scm to manage all of this. They've thrown more at this beast than most companies will throw at their actual product. I'm also not convinced they haven't reinvented…

Your last paragraph doesn't sound like anything at Google. Most engineers will never use branches at all, and even fewer will use branches that merge into trunk (instead of away from it).

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#107

Post by anyfoo » Wed, Jul 25, 2018, 2:29 AM UTC

whack wrote: post #87

anyfoo wrote: post #59

Earlier quoted context omitted.

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" the…

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, al…

> In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it.

The correct course of action is to either reverse/fix the code change to the library you depend on, or if your code is clearly using the library wrong and can be easily fixed, to do that. Not to let the whole ecosystem slowly spiral out of control.

Either way, the point is that it will force the issue to be resolved, quickly, and the code base to move forward.

The tools/libraries you depend on are themselves dependent on other libraries and tools. They may have done changes that are necessary to continue working, which you are not picking up if you stay behind. They will do IPC and RPC and always rely on their infrastructure being current.

>In a monorepo world, you don't have a choice. You've been forcefully migrated

Yes, and that's good, because:

> and migrate only when you want to, and when you're feeling confident about it.

... does not help in moving the code forward.

If your change will break others, you need to coordinate with those others so that the transition happens gracefully, not let them live on what amounts to unsupported (and slowly more incompatible) code.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#108

Post by tiglionabbit » Wed, Jul 25, 2018, 2:32 AM UTC

favorited wrote: post #95

tiglionabbit wrote: post #83

Earlier quoted context omitted.

Fun fact. I asked Facebook why they built their monorepo on Mercurial instead of Git. They said there were scaling issues in Git that made it unusable for large repos and the Git maintainers would not work with them to fix these issues. However, they were able to work with Mercurial to make it capable of holding their entire company in one repo.

Someone from FB did a really cringe-inducing presentation a few years ago about how "X can't handle our scale" (I think the predicate was iOS, but they went into IDEs and SCM systems). They had to pull the video and slides because it was so bad.

XCode also had issues working with large repos. Perhaps they were talking about that?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#109

Post by joshuamorton » Wed, Jul 25, 2018, 2:34 AM UTC

whack wrote: post #62

perfunctory wrote: post #53

Earlier quoted context omitted.

> Working with a monorepo is essentially like the latter. Not really. In the dependencies analogy the author of the dependency has no way to test the dependee(s). While with monorepo this is exactly what you do, "the tooling team" will "carefully test everything" before "propagate into your stack" (and it doesn't have to be irrevocable).

In practice, at any medium/large organization, the tooling team doesn't know your system, and its nuances, nearly well enough to "carefully test everything" . Having a solid automated test suite does help. But I personally would like to be in control of when my project updates its dependencies, instead of being forced to always pull everything from LATEST.

You are, by having tests.

At Google the contract is essentially infrastructure teams (and generally, your dependencies) will not break your unit tests (or will contact you well in advance to handle changes). But if you don't have a test, they might. They don't have to carefully test everything. You do. And if you don't, breakages are entirely your responsibility, because you didn't have a test for them.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#110

Post by hobls » Wed, Jul 25, 2018, 2:37 AM UTC

adamrt wrote: post #69

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

A single team is really helpful. Where I’ve seen it get particularly unhelpful is with multiple teams. I’m also not opposed to the concept, I just think it requires work to do correctly.