Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

91–100 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#91

Post by rossjudson »

andrewfong wrote: post #63
whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

Not saying this is how Google does it, but a monorepo doesn't prevent you from having multiple versions of the same dependency. Ideally, with a monorepo, you could update 99% of your sub-packages to the latest version while still leaving the one alone.

There are a few exceptions to the "one version" rule; the monorepo and the build system support multiple versions just fine. It's just that we don't want them.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#92

Post by whack »

perfunctory wrote: post #76
whack wrote: post #66

Earlier quoted context omitted.

I was referring only to static dependencies, like Guava for example. Static dependencies don't require ongoing "upkeep", so you really should allow your users to use an older version of your library/code, if that's what they really want to do. When it comes to live services that you're actually running on a daily basis, like Google maps, forcing users to migrate makes a lot more sense.

I see. I guess static dependencies are not as prone to ongoing upkeep as services, but they are not totally immune. Think Heartbleed.

Just because your older versions are available to users, doesn't mean you have to continuously maintain them. For example, I know plenty of teams still using Java 7, even though Oracle has EOLed it. The teams know the risk that they are taking, and have the flexibility to decide when they want to migrate to a newer version, and that's how it should be. I would never want to live in a world where Oracle can force me to migrate to Java 10 within a day of release.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#93

Post by mlthoughts2018 »

One of my former managers had worked a long time at Google and was present for the advent of Google’s in-house tooling developed around their monorepo.

His account was that it was basically accidental, at first resulting from short term fire drills, and then creating a snowball effect where the momentum of keeping things in the Perforce monorepo and building tooling around it just happened to be the local optimum, and nobody was interested in slowing down or assessing a better way.

He personally thought working with the monorepo was horrible, and in the company where I worked with him, we had dozens of isolated project repos in Git, and used packaging to deploy dependencies. His view, at least, was that the development experience and reliability of this approach was vastly better than Google’s approach, which practically required hiring amazing candidates just to have a hope of a smooth development experience for everyone else.

I laugh cynically to myself about this any time I ever hear anyone comment as if Google’s monorepo or tooling are models of success. It was an accidental, path-dependent kludge on top of Perforce, and there is really no reason to believe it’s a good idea, certainly not the mere fact that Google uses this approach.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#94

Post by shub »

antt wrote: post #8
ChrisCinelli wrote: post #6

Managing dependencies and versions across repos is a pain. Refactoring across repos is quite hard when your code spreads across repos considering the tree of dependencies. Unfortunately Git checkout all the code, including history, at once and it does not scale to big codebases. The approach that Facebook chose with Mercurial seems a good compromise ( https://code.fb.com/core-data/scaling-mercurial-at-facebook/ )

Git works very well when the code is distributed . Which funnily enough is in the name. That we are using git as a centralized repository is a case of "Why do I need a screwdriver when I have a hammer?".

There's nothing about git that requires you to use it like the kernel does. Centralized version control is just a special case of decentralized, if you're using git. You still get the benefits of your repo being a peer of the master repo, like local branches.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#95

Post by favorited »

tiglionabbit wrote: post #83
acomjean wrote: post #41

Earlier quoted context omitted.

But what are the alternatives to the monorepo in git? All the ways of splitting code up and deploying multiple git repos for one project seem terrible.

Fun fact. I asked Facebook why they built their monorepo on Mercurial instead of Git. They said there were scaling issues in Git that made it unusable for large repos and the Git maintainers would not work with them to fix these issues. However, they were able to work with Mercurial to make it capable of holding their entire company in one repo.

Someone from FB did a really cringe-inducing presentation a few years ago about how "X can't handle our scale" (I think the predicate was iOS, but they went into IDEs and SCM systems). They had to pull the video and slides because it was so bad.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#96

Post by flukus »

robert-brown wrote: post #71
flukus wrote: post #58

Earlier quoted context omitted.

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

Yes, that's what you do. And you commit the change when all the tests pass.

And that's insane, people don't scale like that. It's harder enough keeping your head around one large project let alone every project a company has that you might have to jump into at any point.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#98

Post by maxpert »

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Spot on! I've seen org wide mono repos at Microsoft and they had their custom tooling and build systems built on top of SourceDepot.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#99

Post by hobls »

hobs wrote: post #57
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

heh, thousands, its probably at least an OOM greater, if not two.

Hah, you know I started with millions and then did some fuzzy math and started debating team sizes (since I know google often doesn’t have giant teams), ended up somewhere in the hundreds of thousands, and rounded to thousands. But I made it all caps so you know it’s the SERIOUS kind of thousands. :P
Post reply on HN