Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

11–20 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#11

Post by jsolson »

ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

As mentioned in the post (which is from 2016), Google has also been experimenting with Mercurial as a frontend (in collaboration with "contributors from other companies that value the monolithic source model"). As an avid user of that experiment at Google, it's seems to be going very well.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#13

Post by ridiculous_fish »

> Google's monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully

This 95% number is the most surprising part of the article. That implies that the sum of engineers working on Android + Chrome + ChromeOS + all the Google X stuff + long tail of smaller non-google3 projects (Chromecast, etc) constitute only 5% of their engineers. Is e.g. Android really that small?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#14

Post by ChrisCinelli »

jsolson wrote: post #11
ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

As mentioned in the post (which is from 2016), Google has also been experimenting with Mercurial as a frontend (in collaboration with "contributors from other companies that value the monolithic source model"). As an avid user of that experiment at Google, it's seems to be going very well.

I am not surprised. code.google.com used Mercurial. But I am still curious. Is Mercurial the frontend of Piper or it could live independently? What is open sourced and what is not?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#16

Post by dlubarov »

ridiculous_fish wrote: post #13

> Google's monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully This 95% number is the most surprising part of the article. That implies that the sum of engineers working on Android + Chrome + ChromeOS + all the Google X stuff + long tail of small…

They must have meant that 95% of Google engineers use the monorepo in some capacity, even if the majority of their work is done in a different repo.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#17

Post by dlp211 »

ridiculous_fish wrote: post #13

> Google's monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully This 95% number is the most surprising part of the article. That implies that the sum of engineers working on Android + Chrome + ChromeOS + all the Google X stuff + long tail of small…

I think you're interpretation is incorrect. A better way to think of this is that those 5% of people work exclusively on those projects. I'd be very surprised to learn that only 5% of Google engineers work on those projects.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#18

Post by FartyMcFarter »

jbergknoff wrote: post #15

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of the repo"?

For safe-looking changes, it's OK to only run a subset of the tests (usually including the tests that directly test the changed library).

For changes that are more likely to break distant code, you can run all tests (perhaps bundling together several changes in order not to overload the system).

Alternatively you can take the risk of breaking tests post-submit... this is not very good citizenship, but in some cases it might be reasonable (when the risk is small).

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#19

Post by ebikelaw »

jbergknoff wrote: post #15

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of the repo"?

Dependencies are explicit so the build tool (Bazel, to an approximation) compute the transitive closure of requirements of the desired target.

There are more details about testing at [1]

1: https://static.googleusercontent.com/media/research.google.c...

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#20

Post by guessmyname »

gervase wrote: post #2

Should probably have a [2016] tag.

Indeed, but to be fair, the information in the article is based on several research papers from 2011 [1].

And I am 100% sure the idea of having a monolithic project is several years older than that.

I am grateful that the article is re-posted in multiple websites, because just the other day I was in an interview and, while doing my coding challenge, overheard the conversation of a young computer science graduate and another interviewer. The interviewer asked him to explain what was a monolithic repository and the benefits. This guy had no idea what the interviewer was talking about and right there I realized that what many of us take for granted terminology-wise in the IT world, will certainly be a foreign language to young students who are just entering the work force.

[1] http://info.perforce.com/rs/perforce/images/GoogleWhitePaper...

Post reply on HN