Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

251–260 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#251

Post by wcoenen »

Too wrote: post #245
flukus wrote: post #224

Earlier quoted context omitted.

In the former each project is a self contained unit and if I'm working on project B I can forget project A even exists, which is lovely caused I've got enough to deal with on B as is. Each project can be branched individually, the log of project B is not polluted with commits to project A, I can rebase and not get a bunch of commits I don't care about. The later forces me to be aware of the entire universe in that re…

In the later you can also forget project A exist. git log works in subdirectories out of the box and if you didn't touch any files in project A a rebase will be trivial without any merge conflicts. Branching is also free (unless you are still in the SVN stoneage that requires a copy of each file) so it doesn't matter if you branch the whole monorepo or just a single project.

> unless you are still in the SVN stoneage that requires a copy of each file

Maybe you are thinking of CVS? In SVN, creating branches has always been cheap in both space and time.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#252

Post by thomaslee »

bluejekyll wrote: post #248
thomaslee wrote: post #234

Earlier quoted context omitted.

Exactly this. Or at least it's a way this can be achieved, assuming solid testing & some tooling in the mix. For folks unfamiliar with it, the issue is something like: 1. You find a bug in a library A. 2. Libraries B, C and D depend on A. 3. B, C and D in turn are used by various applications. How do you fix a bug in A? Well, "normal" workflow would be something like: fix the bug in A, submit a PR, wait for a CI buil…

> So having run into this problem, folks like Google, Twitter, etc. use monorepos to help address some of this. I think you’re retroactively claiming that Google actively anticipated this in their choice at the beginning of using Perforce as an SCM. They may believe that it’s still the best option for them, but as I understand it, to make it work they bought a license to the Perforce source code forked it and practic…

> I think you’re retroactively claiming that Google actively anticipated this in their choice at the beginning of using Perforce as an SCM.

Oh I didn't mean to imply exactly that, but really good point. I just meant that it seems like folks don't typically _anticipate_ these issues so much as they're forced into it by ossifying velocity in the face of sudden wild success. I know at least a few examples of this happening -- but you're right, those folks were using Git.

In Google's case, maybe it's simply that their centralized VCS led them down a certain path, their tooling grew out of that & they came to realize some of the benefits along the way. I'd be interested to know too. :)

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#254

Post by HNNewer »

therealmarv wrote: post #253
IloveHN84 wrote: post #162

The giant monorepo works only if you're using SVN, with Git it would be tremendous

Unless you change git like Microsoft did: https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...

sure, but GitHub / GitLab / Bitbucket don't offer it

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#255

Post by trasz »

dekhn wrote: post #79
pcwalton wrote: post #32

Earlier quoted context omitted.

There's also the fact that monorepos have issues when you don't have one organization responsible for all the code. The Linux kernel and NetHack don't live in the same repository for good reason.

I dunno, the BSD distribution included a wide gamut of games along with the kernel source in the same tree. In fact, NetHack is derived from Hack which itself is derived from Rogue, which was distributed within BSD. And BSD represented a cross-organization responsibility (see the history of AT&T and BSD).

Wasn't it because it was the same group of people who worked on both? And when it ceased to make sense, the games were split off - the only remaining ones in FreeBSD are things like banner(6) or pom(6).

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#256

Post by trasz »

int_19h wrote: post #225
flukus wrote: post #205

Earlier quoted context omitted.

> There is no "transitional" stage, when the kernel is already patched, but no user space apps are ready for those changes yet. It all happens at once. What about third party apps? It's not a fully self contained system, there are binaries out there running on openBSD that the openBSD devs have never heard of, and they were broken by the change.

BSDs simply don't guarantee ABI stability, so no third party app should ever make a syscall directly. It all goes via libc. So, yes, from that perspective, it is a fully self-contained system. In practice, third-party apps sometimes think that they know better, and get broken. Anything written in Go, for example: https://github.com/golang/go/issues/16272

Sure they do guarantee the ABI stability (within major release), but the main thing here is that FreeBSD - like pretty much any other operating system, but differently from Linux - maintains the stability at the libc level, not at syscall level.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#257

Post by paulddraper »

jgibson wrote: post #194

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have…

> The two don't have to be combined.

They do have to be combined in some way, at least to be reproducible. Your requirements.txt example is one way of combining version control + dependencies: give code an explciit version and depend on it elsewhere by that version.

Google has chosen to do combine them in a different way, where ever commit of a library implicitly produces a new version, and all downstream projects use that.

> googles internal tool handle permissions on a granular basis?

Not sure what you mean...it's build tool handles package visibilty (https://docs.bazel.build/versions/master/be/common-definitio...). It's version control tool handles edit permissions (https://github.com/bkeepers/OWNERS).

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#258

Post by paulddraper »

adrianN wrote: post #250
justicezyx wrote: post #220

Earlier quoted context omitted.

Single repo is one design that coherently addresses source control management and dependency management. The key is to let the repo be a single comprehensive source of data for building arbitrary artifacts.

A single repo makes it a bit tricky to use some library in version A for project X and version B for project Y.

Correct.

You can consider that a bad thing or a good thing.

Most language's package composition (C/C++, Java, Python, Ruby) don't permit running multiple versions at runtime. The single-version policy is one way of addressing dependency hell.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#259

Post by paulddraper »

Version controlled repositories are like business offices.

You can have your entire company in one location, or the entire company in separate locations. The most important thing is the logical rather than physical organization: team structure, executive leadership, inter-org dependencies, etc. You can achieve autonomy and good structure with or without separate locations.

A single location reduces barriers, but at some point multiple locations can solve physical and logistical challenges. General rule of thumb is to own and operate office space in a few locations as possible, but at some point you have to take drastic measures one way or another.

(Notice that Google had to invent their own proprietary version control system just for their monorepo. And not even Google actually uses a single repo as the source of truth: e.g. Chromium and Android.)

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#260

Post by paulddraper »

wirrbel wrote: post #153
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Same line of thinking, just different conclusions. I feel terrible for anyone trying to run a company with open-source style independent repos. On a popular github project, you have MANY potential contributors that will tell you if a PR, or a release candidate break API compatibility, etc. There are thousands of hours in open source dedicated to fixing integration issues due to the (unavoidable) poly-repo situation.…

> It is split up into libraries (or microservices)

It's a frequent problem to conflate organization/modularization with lifecycle/version management.

You can have a well-organized codebase just as easily in a monorepo.

That's a separate question from management the lifecycle of the code. (What is release and when? What tests are run? What process approves a change?)

Post reply on HN