Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

141–150 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#141

Post by artursapek »

adamrt wrote: post #69
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

We had this same experience with a large golang project, consisting of about 8 individual services. Switching to a monorepo made it incredibly easy to make changes to common code and inter-service communication. Huge breath of fresh air.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#142

Post by malkia »

testcross wrote: post #116
malkia wrote: post #101

Earlier quoted context omitted.

What's missing in these is cross-reference, which is not possible without somewhat established BUILD system (caps "pun-intened") - e.g. like bazel/build, then a source code indexer, etc, etc. This becomes very critical for doing reviews, since it allows you to "trace" things without running them, apart from many other things. For example large scale refactorings looking for usages of functions, and other examples lik…

They can create a standard file format that has to be generated by build system. github is in a pretty powerful position. They can create even a shitty version of it and people will follow. I've been thinking about a tool like this for a long time. A way to attach to each commit not only the diff in the code, but also the list of places affected by the changes (usages of functions that are modified for example). Then…

Even if they can, it's one thing indexing your own source files every night, another indexing a much bigger amount + massive amounts of branches, clones, etc. (I'm talking about github) - e.g. not practical - as there is no no clear way to say which branch (from git) must be indexed (obviously not all) - e.g. there is no encompassing "standard" saying so.

That by itself is another BIG PLUS for mono-repo (and "mono"-rules) - things are done one (opinionated) way, trunk based development - but thus giving you things that you won't be able to have normally.

Now indexing source file is not an easy and cheap task - it's basically a huge MapReduce done over several hours (just guessing), so there must be a reason for this to be done.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#143

Post by tzhenghao »

Having worked at different companies adopting both monorepo and the multiple repos approach, I find monorepo a better normalizer at scale in consolidating all "software" that runs the company.

Just like what many commenters here have mentioned, the monorepo approach is a forcing function on keeping compatibility issues at bay.

What you don't want is to end up in a situation where teams reinvent their own wheels instead of building on top of existing code, and at scale, I think the multiple repo approach tends to breed such codebase smell. [1] I'm sure 8000 repos is living hell for most organizations.

[1] - https://www.youtube.com/watch?v=kb-m2fasdDY

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#144

Post by carapace »

https://en.wikipedia.org/wiki/Conway%27s_law

> "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."

Interestingly, in light of the above adage, this massive repo is organized (if that's the word for it) like a bazaar or flea market. (Rather than like a phone book https://en.wikipedia.org/wiki/Yellow_pages )

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#145

Post by fizixer »

I don't care about that. For me this is incomprehensible:

Why the eff does Google have billions of lines of code in their repo?

I hope they are not counting revisions (e.g., if a single 1 million project has 100 revisions, that's 1 million, not 100 million).

I have heard that they do count generated code (so it's not all handwritten code). In that case again, I have two things to say:

- that's a bad metric. I could overnight generate a billion lines of code with each line a printf of number_to_word of numbers from 1 to a billion. They want to measure the size of the repo? They should tell us the gigabytes, terabytes etc. But when it's lines of code, it's cheezy and childish to blow up the measure by including lines of generated code.

- But more importantly, I hope the generated code is 90% or more of that repository. Because any less than that would mean that Google engineers have handwritten 100 million or more lines of code through out the lifetime of the company, in which case I have to ask: what bloated mess do you have on your hands? I thought you guys were the top engineers of the world.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#146

Post by georgewfraser »

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I kinda disagree, we’re a dev team of 30, 3.5 years in, 150k lines of code and we’ve always had a monorepo. We had to switch from maven to bazel after about 2 years because test times got out of control; bazel has been about 50% more annoying than maven but the incremental builds work perfectly.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#147

Post by ebikelaw »

Tloewald wrote: post #136
adamrt wrote: post #69

Earlier quoted context omitted.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

My previous had a monorepo for the website and backend (but not the mobile apps) which was insane to work with (as a coder I had a dedicated 128 core box to work on, some engineers had more than one, less intense engineers shared one) and a substantial amount of my time was spent just finding code. I guess most engineers just end up working in some nook and so that searching code constantly becomes less of an issue (…

You seem to have conflated a bunch of different things and confused them with a monolithic repository. There's no reason why a monorepo requires you to have a gigantic development box ... you identify and compile only the transitive dependencies of your target, not every line of code in the repo.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#148

Post by mcny »

Tloewald wrote: post #136
adamrt wrote: post #69

Earlier quoted context omitted.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

My previous had a monorepo for the website and backend (but not the mobile apps) which was insane to work with (as a coder I had a dedicated 128 core box to work on, some engineers had more than one, less intense engineers shared one) and a substantial amount of my time was spent just finding code. I guess most engineers just end up working in some nook and so that searching code constantly becomes less of an issue (…

I recently saw a question on quora asking whether the free food at Google boosted productivity. The reply that seemed strange at thwt time was being able to focus on their job and not having to do a bunch of stuff is what boosted productivity at Google.

I get some perspective from the comments above. There is seemingly an army of engineers at Google that keeps the monorepo functioning. I was at a meeting about bazel and angular. I thought I'd ask how they do things at Google. To my surprise, the presenter said he is not at liberty to discuss how things work at Google. I guess it wasn't so surprising in hindsight. I mean what would I do with that information, right? It would be way too overkill for my tiny crud application.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#149

Post by bedros »

Tloewald wrote: post #130
bedros wrote: post #112

Earlier quoted context omitted.

how do you create branches in mono repo? for example I want to use branch rev5 from project A and rev3 from project B how I do that in a mono repo, I could not do it in HG, but sure about GIT

Can’t you create a branch and merge the two branches you are interested in into that?

my understanding, if you branch, you branch the entire repo, (not sure about some special case extensions ) if you have two projects stored in a single repo, you are forced to use whatever rev at for each project at a point of branch rev 5543 for example

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#150

Post by iamforreal »

shados wrote: post #132
fcarraldo wrote: post #123

Earlier quoted context omitted.

People also seem to forget that "Monorepo" or (many) "Microrepos" is not a binary choice. You can have both tiny repositories which do a single thing and large repositories that consist of many projects. It's totally cool to have both, assuming your team can be trusted to make the appropriate choices as they create new projects. > And then people forget how to make distributed repos work and claim things like "omg I…

> Is this a solved problem I don't mean that it's magical, just that it's not particularly sorcery. Instead of making a breaking change, add new method, deprecate old method. Update projects, then get rid of old deprecated method. Because they're distinct you can do this one by one so some project can reap the benefits without having to wait until all the problems are solved. Some people in this thread act like its f…

You can also just have tooling to find every reference of a function and then refactoring all at once, sending it out in a single pr, but that's a bit more advanced
Post reply on HN