Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

61–70 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#61

Post by rco8786 »

hobls wrote: post #49
rco8786 wrote: post #34

Earlier quoted context omitted.

Are you suggesting that there’s a solution to managing large amounts of code that doesn’t involve large amounts of tooling?

There’s an important distinction between lots of code and lots of projects. I agree; if you have a ton of code you’d better invest in tooling. But if you just have several normally sized projects, a monorepo can make your life much more difficult than simply using several repos.

Sure. You could also replace the term “monorepo” with “separate repos” and your statement would be just as valid. Either way you go has pros and cons.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#62

Post by whack »

perfunctory wrote: post #53
whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

> Working with a monorepo is essentially like the latter. Not really. In the dependencies analogy the author of the dependency has no way to test the dependee(s). While with monorepo this is exactly what you do, "the tooling team" will "carefully test everything" before "propagate into your stack" (and it doesn't have to be irrevocable).

In practice, at any medium/large organization, the tooling team doesn't know your system, and its nuances, nearly well enough to "carefully test everything".

Having a solid automated test suite does help. But I personally would like to be in control of when my project updates its dependencies, instead of being forced to always pull everything from LATEST.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#63

Post by andrewfong »

whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

Not saying this is how Google does it, but a monorepo doesn't prevent you from having multiple versions of the same dependency. Ideally, with a monorepo, you could update 99% of your sub-packages to the latest version while still leaving the one alone.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#64

Post by prepend »

I love these articles. Is there a wiki or collection of detailed descriptions of large company tech practices that isn’t marketing blargh.

I read years ago about Google data ingest, locator process but neglected to bookmark so now can’t find the reference.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#65

Post by perfunctory »

whack wrote: post #62
perfunctory wrote: post #53

Earlier quoted context omitted.

> Working with a monorepo is essentially like the latter. Not really. In the dependencies analogy the author of the dependency has no way to test the dependee(s). While with monorepo this is exactly what you do, "the tooling team" will "carefully test everything" before "propagate into your stack" (and it doesn't have to be irrevocable).

In practice, at any medium/large organization, the tooling team doesn't know your system, and its nuances, nearly well enough to "carefully test everything" . Having a solid automated test suite does help. But I personally would like to be in control of when my project updates its dependencies, instead of being forced to always pull everything from LATEST.

> the tooling team doesn't know your system

they don't know it because they don't use monorepo. monorepo makes "solid automated tests" easier since basically there is only one version to test. The instinct against pulling everything from LATEST, developed in the traditional world, is perfectly understandable. However in monorepo "your" project is also tooling-team's project. "being forced" becomes "being helped". It's shared responsibility.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#66

Post by whack »

perfunctory wrote: post #60
whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

>In a versioned world, you can release your change as a new version, and allow your users to self-select Repeat this process multiple times and you end up with configuration/settings hell. Been there done that. It's not black and white but "trampling over the 0.1%" could be a sensible business/architectural decision. For example how do you imagine "google maps" users selecting when/how to migrate?

I was referring only to static dependencies, like Guava for example. Static dependencies don't require ongoing "upkeep", so you really should allow your users to use an older version of your library/code, if that's what they really want to do.

When it comes to live services that you're actually running on a daily basis, like Google maps, forcing users to migrate makes a lot more sense.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#68

Post by dmoy »

threeseed wrote: post #45
nine_k wrote: post #36

Earlier quoted context omitted.

Most folks who consider a monorepo don't have billions of lines of code, and often not even millions. Linux kernel is a monorepo.

Linux kernel is one functional piece of work though. Imagine if we combined KDE, Gnome, Linux Kernel, ZFS etc all in the one monorepo.

And gnucash, libreoffice, a couple copies of android, three other things that forked the linux kernel, and then all of apache to boot.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#69

Post by adamrt »

hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

We moved to a monorepo about 2 years ago and it has been nothing but success for us.

We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all development apps are communicating with each other. I don't think we've had any issues that I can recall.

We are a reasonably small team though, so maybe that is part of it.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#70

Post by vkou »

flukus wrote: post #58
ebikelaw wrote: post #52

Earlier quoted context omitted.

See, this is where your argument broke down for me. Once you’ve decided there is some library of common code, and assuming you factor out that code into another repo, you’ve just lost your ability to easily make breaking changes to the common code, which is something trivially easy to do in a monorepo. Why would you want that? It seems to me that if you have multiple projects sharing a base of common code then a mono…

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

You can soft-deprecate the old code path, and communicate that warnings will turn into errors at some future date.

You can send your pull request to the affected team leads, and request that they approve it, once they make changes on their end.

I mean, the alternative is that you have 17 different projects, each using one of five different versions of the common code. Heaven forbid one of them makes an incorrect assumption about another. Getting 17 different teams to dance together around a breaking change is always going to be hard.

Post reply on HN