Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

241–250 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#241

Post by akvadrako »

dekhn wrote: post #79
pcwalton wrote: post #32

Earlier quoted context omitted.

There's also the fact that monorepos have issues when you don't have one organization responsible for all the code. The Linux kernel and NetHack don't live in the same repository for good reason.

I dunno, the BSD distribution included a wide gamut of games along with the kernel source in the same tree. In fact, NetHack is derived from Hack which itself is derived from Rogue, which was distributed within BSD. And BSD represented a cross-organization responsibility (see the history of AT&T and BSD).

It does seem like the Linux model scales better than the BSD model.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#242

Post by tom_ »

bedros wrote: post #112
adamrt wrote: post #69

Earlier quoted context omitted.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

how do you create branches in mono repo? for example I want to use branch rev5 from project A and rev3 from project B how I do that in a mono repo, I could not do it in HG, but sure about GIT

Branch rev5 of project A's folder into a folder in your project's folder, branch rev3 of project B's folder into a folder in your project's folder. Get your project to refer to its own copy, rather than the shared copies you branch from. (This isn't quite like git submodules, but that's probably the closest thing git has.)

You do the usual 3-way merge thing to push your changes upstream, or pull upstream changes into your copy. As with git, the VCS tracks which revision of upstream your copy is up to date with, which is how it determines the base for the 3-way merge.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#243

Post by mrweasel »

013a wrote: post #173
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I call this "Google Imposter Syndrome". Because Google (insert Facebook, Apple, Amazon, etc) has success with Monorepos (insert gRPC, Go, Kubernetes, React/Native, etc), it must be a great idea, we should do it. You see this everywhere . Also known as an Appeal to Authority. My personal opinion: very few companies will hit a point where sheer volume of code or code changes makes a monorepo unwieldy. Code volume is a…

Most people/companies aren't Google, but as you say they assume that if one or more of the tech gigants (or other very public tech companies) are doing something, then it must be good.

Often the focus is extremely weird. When people noticed that WhatsApp only employed something like 45 engineers, then most assumed that it was because they used Erlang and FreeBSD. The thought that maybe their success was do to hiring the very best engineers and paying accordingly is less attractive.

Monorepos is just a another item to the heap of things that may be a good idea, but it depends.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#244

Post by joe_fishfish »

This is probably a stupid question, but I couldn't find an answer. Does this mean Google keeps all of its different products in all their different languages and environments in one repo? So like, Android lives in the same repo as Gmail, which is the same repo as all the Waymo code and the Google search engine code as well? That seems insane to me.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#245

Post by Too »

flukus wrote: post #224
Too wrote: post #212

Earlier quoted context omitted.

How is this company /ProjectA /.git /ProjectB /.git easier to browse than this? company /.git /ProjectA /ProjectB You still need to find the project A repo if you don't use monorepos. And even if you do use monorepos everything doesn't have to be one monoloithic build hogging down your IDE, you can still have microservices with the code for each hosted in the same repo. You seem to conflate monorepo with lots of othe…

In the former each project is a self contained unit and if I'm working on project B I can forget project A even exists, which is lovely caused I've got enough to deal with on B as is. Each project can be branched individually, the log of project B is not polluted with commits to project A, I can rebase and not get a bunch of commits I don't care about. The later forces me to be aware of the entire universe in that re…

In the later you can also forget project A exist. git log works in subdirectories out of the box and if you didn't touch any files in project A a rebase will be trivial without any merge conflicts. Branching is also free (unless you are still in the SVN stoneage that requires a copy of each file) so it doesn't matter if you branch the whole monorepo or just a single project.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#246

Post by p-schultz »

joe_fishfish wrote: post #244

This is probably a stupid question, but I couldn't find an answer. Does this mean Google keeps all of its different products in all their different languages and environments in one repo? So like, Android lives in the same repo as Gmail, which is the same repo as all the Waymo code and the Google search engine code as well? That seems insane to me.

Yes, exactly. The self-driving car is in there too.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#247

Post by csdreamer7 »

justincormack wrote: post #31
csdreamer7 wrote: post #9

Earlier quoted context omitted.

Doesn't the Git Virtual File system that Microsoft is contributing to Git take care of this? https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin... Edit: don't just down vote. If you have a problem with my comment, tell me why.

This currently only works on Windows, although they are planning OSX and Linux ports.

That is important. My link was back in 2017. Do you know of any ETA for GVFS getting on Linux?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#248

Post by bluejekyll »

thomaslee wrote: post #234
edejong wrote: post #222

Earlier quoted context omitted.

The key here is reverse dependency management. “If I change X, what would influence this change?”. This can be achieved with single repo better than multi-repo due to the completeness of the (dependency) graph.

Exactly this. Or at least it's a way this can be achieved, assuming solid testing & some tooling in the mix. For folks unfamiliar with it, the issue is something like: 1. You find a bug in a library A. 2. Libraries B, C and D depend on A. 3. B, C and D in turn are used by various applications. How do you fix a bug in A? Well, "normal" workflow would be something like: fix the bug in A, submit a PR, wait for a CI buil…

> So having run into this problem, folks like Google, Twitter, etc. use monorepos to help address some of this.

I think you’re retroactively claiming that Google actively anticipated this in their choice at the beginning of using Perforce as an SCM. They may believe that it’s still the best option for them, but as I understand it, to make it work they bought a license to the Perforce source code forked it and practically rewrote it to work.

Here’s a tech talk Linus gave at Google in 2007: https://youtu.be/4XpnKHJAok8

My theory (I wonder if someone can confirm this), is that Google was under pressure at that point with team size and Perforce’s limitations. Git would have been an entirely different direction had they chosen to ditch p4 and instead use git. What would have happened in the Git space earlier if that had happened? Fun to think about... but maybe Go would have had a package manager earlier ;)

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#249

Post by w_t_payne »

mehh wrote: post #213
w_t_payne wrote: post #195

Earlier quoted context omitted.

We have about 400 repos in a team of about 20 developers. We do have extensive tooling to help coordinate all of these, but configuration management is still by far the biggest engineering challenge that we face. I don't recall having such issues when I was working with Subversion and Perforce. On the other hand, not everything was rosy in the 'good old days': MS Source Safe was (by far) the worst VCS experience that…

Sounds excessive and beyond the norms of what most people would encounter in not having a mono repo i.e. the opposite extreme.

Yeah. It is a bit excessive, and it is a PITA to manage.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#250

Post by adrianN »

justicezyx wrote: post #220
jgibson wrote: post #194

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have…

Single repo is one design that coherently addresses source control management and dependency management. The key is to let the repo be a single comprehensive source of data for building arbitrary artifacts.

A single repo makes it a bit tricky to use some library in version A for project X and version B for project Y.
Post reply on HN