Why Google Stores Billions of Lines of Code in a Single Repository (2016) • Hacker News Board

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#191

Post by w_t_payne » Wed, Jul 25, 2018, 6:25 AM UTC

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Tooling is required for coordinating configuration management on multiple repositories too.

Also, why isn't such tooling available as open source? I'm trying to do my bit, but we could do with more effort being put into this, somehow.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#192

Post by zuppy » Wed, Jul 25, 2018, 6:27 AM UTC

wirrbel wrote: post #153

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Same line of thinking, just different conclusions. I feel terrible for anyone trying to run a company with open-source style independent repos. On a popular github project, you have MANY potential contributors that will tell you if a PR, or a release candidate break API compatibility, etc. There are thousands of hours in open source dedicated to fixing integration issues due to the (unavoidable) poly-repo situation.…

We have shared components between some of our projects, I'm not sure how monorepo will fit in here. For integration there are a lot of available build tools and repo management apps. For us it solves the problem of having no dependency between versions of the same library used between multiple products.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#193

Post by not_kurt_godel » Wed, Jul 25, 2018, 6:31 AM UTC

ebikelaw wrote: post #52

flukus wrote: post #48

Earlier quoted context omitted.

> All the ways of splitting code up and deploying multiple git repos for one project seem terrible. Of course they are, git isn't the tool for this. You don't want multiple repos for a single project, you want one per project (this is not a monorepo). If there are things like code common to multiple projects then they are their own project with their own repo and release schedule, releases go into some sort of packag…

See, this is where your argument broke down for me. Once you’ve decided there is some library of common code, and assuming you factor out that code into another repo, you’ve just lost your ability to easily make breaking changes to the common code, which is something trivially easy to do in a monorepo. Why would you want that? It seems to me that if you have multiple projects sharing a base of common code then a mono…

> you’ve just lost your ability to easily make breaking changes to the common code

It should be hard to make breaking changes in common code. Even 'trivial' breaking changes seem to have a way of breaking things even when they shouldn't. If you need to make a breaking change to common code, the proper way to do it is add the new functionality separately, deprecate the old functionality (i.e. with javadoc so it gets called out explicitly by the IDE), and incorporate it 1-by-1 into consumers until none are using the deprecated version anymore.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#194

Post by jgibson » Wed, Jul 25, 2018, 6:32 AM UTC

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have them in the same repository, but still have Project X list its dependency of project Y in a requirements.txt file at a particular version. With the last option, you get the benefit of mono-repo tooling (easier search, versioning, etc) but you can control your own dependencies if you want.

edit: I do have one question though, does googles internal tool handle permissions on a granular basis?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#195

Post by w_t_payne » Wed, Jul 25, 2018, 6:33 AM UTC

shados wrote: post #132

fcarraldo wrote: post #123

Earlier quoted context omitted.

People also seem to forget that "Monorepo" or (many) "Microrepos" is not a binary choice. You can have both tiny repositories which do a single thing and large repositories that consist of many projects. It's totally cool to have both, assuming your team can be trusted to make the appropriate choices as they create new projects. > And then people forget how to make distributed repos work and claim things like "omg I…

> Is this a solved problem I don't mean that it's magical, just that it's not particularly sorcery. Instead of making a breaking change, add new method, deprecate old method. Update projects, then get rid of old deprecated method. Because they're distinct you can do this one by one so some project can reap the benefits without having to wait until all the problems are solved. Some people in this thread act like its f…

We have about 400 repos in a team of about 20 developers. We do have extensive tooling to help coordinate all of these, but configuration management is still by far the biggest engineering challenge that we face.

I don't recall having such issues when I was working with Subversion and Perforce.

On the other hand, not everything was rosy in the 'good old days': MS Source Safe was (by far) the worst VCS experience that I have ever had.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#196

Post by aidos » Wed, Jul 25, 2018, 6:44 AM UTC

bedros wrote: post #112

adamrt wrote: post #69

Earlier quoted context omitted.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

how do you create branches in mono repo? for example I want to use branch rev5 from project A and rev3 from project B how I do that in a mono repo, I could not do it in HG, but sure about GIT

Depends on the system. When I used to manage SVN we would branch independent projects and then releases would be a snapshot of each into the server section of the repo. Those were then pulled down to their respective machines.

In SVN a branch is simply a convention. You copy (almost zero cost) things around into your branches directory

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#197

Post by AmericanChopper » Wed, Jul 25, 2018, 6:48 AM UTC

shados wrote: post #114

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Both monorepo or "micro repo" end up falling apart at scale without some devops work involved. Either will work if you only have a few dozen projects. Neither will work once you hit 10s of millions of lines of code. But people seem to forget that it wasn't that long ago that git didn't exist, making multiple repos was a pain in the butt. Managing multiple repos locally was hell. Monorepos were the norm. Then as the s…

>Both monorepo or "micro repo" end up falling apart at scale without some devops work involved

Wouldn’t any project fall apart without devops work?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#198

Post by senozhatsky » Wed, Jul 25, 2018, 6:56 AM UTC

glandium wrote: post #160

senozhatsky wrote: post #81

Well, it's not so uncommon. For instance, OpenBSD, NetBSD repos are sort of monolithic. And, believe it or not, there are some advantages. For instance, let's take a look at OpenBSD 5.5 [0] release notes: > OpenBSD is year 2038 ready and will run well > beyond Tue Jan 19 03:14:07 2038 UTC OpenBSD 5.5 was released on May 1, 2014. While Linux is still "not quite there yet" y2038-wise. y2038 is a very complex issue, whi…

The reason why y2038 problem looked so simple to OpenBSD has little to do with "monolithic repository" and everything to do with "happy to break kernel ABI compatibility". You're saying as much yourself. Monolithic repository might have been a tool that helped enforce it, but that's not what made it happen. It's the decision that ABI could be broken that did. And that's also why it hasn't happened in Linux yet. Even…

> Monolithic repository might have been a tool that helped enforce it, > but that's not what made it happen. It's the decision that ABI could > be broken that did.

Well, there are probably some subtle details which I'm missing, and may be you are totally right.

The way it looks to me is as follows: They are "happy to break kernel ABI compatibility" because the repository is monolithic - they break ABI, they immediately fix user space apps.

E.g. NetBSD time_t 64-bit commit: https://marc.info/?l=openbsd-cvs&m=137637321205010&w=2

They patched the kernel:

	 sys/kern       : kern_clock.c kern_descrip.c kern_event.c
	                 kern_exit.c kern_resource.c kern_subr.c 
	                 kern_synch.c kern_time.c sys_generic.c 
	                 syscalls.conf syscalls.master vfs_getcwd.c 
	                 vfs_syscalls.c vfs_vops.c

and fixed broken user space at the same time:

...

	 sys/msdosfs    : msdosfs_vnops.c
	 sys/netinet6   : in6.c nd6.c
	 sys/nfs        : nfs_serv.c nfs_subs.c nfs_vnops.c xdr_subs.h
	 sys/ntfs       : ntfs_vnops.c
	 sys/sys        : _time.h _types.h dirent.h event.h resource.h
	                 shm.h siginfo.h stat.h sysctl.h time.h types.h 
	                 vnode.h 
	sys/ufs/ext2fs : ext2fs_lookup.c 
	sys/ufs/ufs    : ufs_vnops.c

...

There is no "transitional" stage, when the kernel is already patched, but no user space apps are ready for those changes yet. It all happens at once.

-ss

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#199

Post by golangnews » Wed, Jul 25, 2018, 6:57 AM UTC

mr_tristan wrote: post #42

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I sense that Google invests much more in it's infrastructure then most companies make in revenue. I've worked with monorepos, and I'd be loathe to recommend it as well; the combination of culture shift and tooling it takes to keep a monorepo system running makes most CD processes you see today look like child's play. There is a lot of very good free software that supports most of the open source approach to CD these…

What problems specifically did you see? Was this because the repo was too large?

I understand at google scale you'd need lots of tooling but why at a smaller scslr of merging a dozen small repos?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#200

Post by perlgeek » Wed, Jul 25, 2018, 6:57 AM UTC

senozhatsky wrote: post #81

Well, it's not so uncommon. For instance, OpenBSD, NetBSD repos are sort of monolithic. And, believe it or not, there are some advantages. For instance, let's take a look at OpenBSD 5.5 [0] release notes: > OpenBSD is year 2038 ready and will run well > beyond Tue Jan 19 03:14:07 2038 UTC OpenBSD 5.5 was released on May 1, 2014. While Linux is still "not quite there yet" y2038-wise. y2038 is a very complex issue, whi…

... and all the third-party software that was compiled for older versions of OpenBSD is now also broken by default.

The problem is that this approach only works if it is really a self-contained system. But OpenBSD isn't: it's a basis to run software, potentially third-party software. It's can't be a closed Universe and still be useful at the same time.