Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

21–30 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#21

Post by dekhn »

jbergknoff wrote: post #15

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of the repo"?

You specify targets. Just like using bazel: bazel build //tensorflow/blah/....

I maintain a small part of the monorepo, and it's really nice to be able say "Run every test that transitively depends on numpy with my uncommitted changes", so you can know if your changes break anybody who uses numpy when you update the version.

Personally I think it would be neat if there was an external "virtual monorepo" that integrated as-close-to-head of all software projects (starting at the root, that's things like absl and icu, with the tails being complex projects like tensorflow), and constantly ran CI to update the base versions of things. Every time I move to the open source world, I basically have to recompile the world from scratch and it's a ton of work.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#22

Post by dlubarov »

jbergknoff wrote: post #15

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of the repo"?

It's flexible; presubmit tests can be configured per-directory. There's also an option to run all tests of packages that could be affected by a change based on the Blaze dependency graph.

If you're making changes to a package with tons of dependencies such as Guava, for a risky change you might want to run all affected tests, but for a minor change you might want to run just the standard unit tests. As a compromise, there's also an option to run a random sample of affected tests.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#23

Post by mason55 »

gravypod wrote: post #5
curtis wrote: post #4

I think monorepos make a lot of sense when you're talking about millions of lines of code. I'm not at all sure they make sense when you're talking about billions.

I don't think the number of linea matters. I think the interconnection of your code matters. If you have 2 sets of services that are completely uncoupled the having two monorepos for those two deployments make sense. If you can guarantee atomic changes across all services that interconnect you have the benefits monorepos give you.

Isn’t this only true if you’re doing full CI? Otherwise I could update my service and you can update yours to work with mine but unless we coordinate deployments you still have to worry about interface mismatches. I guess the alternative is you can just never (for a loose definition of never) make breaking changes to an interface. You can only enhance or create a new version.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#25

Post by hyperpape »

ridiculous_fish wrote: post #13

> Google's monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully This 95% number is the most surprising part of the article. That implies that the sum of engineers working on Android + Chrome + ChromeOS + all the Google X stuff + long tail of small…

I don’t know how to parse the number, but 5% of a billion still leaves 50 million lines of code, or three Linux kernels worth.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#26

Post by Karishma1234 »

ridiculous_fish wrote: post #13

> Google's monolithic software repository, which is used by 95% of its software developers worldwide, meets the definition of an ultra-large-scale4 system, providing evidence the single-source repository model can be scaled successfully This 95% number is the most surprising part of the article. That implies that the sum of engineers working on Android + Chrome + ChromeOS + all the Google X stuff + long tail of small…

The 95% number probably does not mean what you are saying but what it means is 95% of developers are using it for some reason with say a non-zero commits over lifetime of a developer OR simply checking it out and using it for dependancies.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#27

Post by kajecounterhack »

ChrisCinelli wrote: post #14
jsolson wrote: post #11

Earlier quoted context omitted.

As mentioned in the post (which is from 2016), Google has also been experimenting with Mercurial as a frontend (in collaboration with "contributors from other companies that value the monolithic source model"). As an avid user of that experiment at Google, it's seems to be going very well.

I am not surprised. code.google.com used Mercurial. But I am still curious. Is Mercurial the frontend of Piper or it could live independently? What is open sourced and what is not?

Mercurial is used as a frontend to piper in that experiment. It doesn't live independently. Piper isn't open source.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#28

Post by hobls »

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#29

Post by [deleted] »

ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

[deleted]
Post reply on HN