Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

31–40 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#31

Post by justincormack »

csdreamer7 wrote: post #9
ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

Doesn't the Git Virtual File system that Microsoft is contributing to Git take care of this? https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin... Edit: don't just down vote. If you have a problem with my comment, tell me why.

This currently only works on Windows, although they are planning OSX and Linux ports.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#32

Post by pcwalton »

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

There's also the fact that monorepos have issues when you don't have one organization responsible for all the code. The Linux kernel and NetHack don't live in the same repository for good reason.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#33

Post by jorblumesea »

Is this really relevant for anyone except for "google scale" companies? For most teams, managing 30-40 services backed by git repos isn't a huge task and doesn't cause many problems.

Is there mature tooling that helps teams manage this, or is this proprietary google magic tooling?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#34

Post by rco8786 »

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Are you suggesting that there’s a solution to managing large amounts of code that doesn’t involve large amounts of tooling?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#36

Post by nine_k »

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Most folks who consider a monorepo don't have billions of lines of code, and often not even millions.

Linux kernel is a monorepo.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#37

Post by jldugger »

curtis wrote: post #4

I think monorepos make a lot of sense when you're talking about millions of lines of code. I'm not at all sure they make sense when you're talking about billions.

Well, this particular monorepo has two billion LoC. But it's not a git monorepo, which matters significantly.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#38

Post by bluedino »

ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

>> Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases

A shallow clone can be helpful in cases like this

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#39

Post by fastball »

jorblumesea wrote: post #33

Is this really relevant for anyone except for "google scale" companies? For most teams, managing 30-40 services backed by git repos isn't a huge task and doesn't cause many problems. Is there mature tooling that helps teams manage this, or is this proprietary google magic tooling?

Most teams can probably get by with much fewer than 30-40 services. Unless you have 30-40 groups within your team.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#40

Post by EpicEng »

antt wrote: post #8
ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

Git works very well when the code is distributed . Which funnily enough is in the name. That we are using git as a centralized repository is a case of "Why do I need a screwdriver when I have a hammer?".

> Managing dependencies and versions across repos is a pain
Post reply on HN