Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

121–130 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#121

Post by jmillikin »

taeric wrote: post #116
jcranmer wrote: post #88

Earlier quoted context omitted.

So why is this a problem for a monorepo but not the multi-repo? It seems to me that the major difference is that in a multi-repo, you'd be more likely to be oblivious to the multitude of dependency issues than you are in the monorepo... and to be honest, that actually sounds like a bad thing to me, because it means you're sweeping legitimate issues underneath the rug.

In a multi-repo, you don't build source dependencies between projects. You can do this with a mono, as well. However, the conceit is that "in the same repo" means you can "change them together." It is very very tempting that "went out as a single commit" means that it went out fine. Which, just isn't something you see in a multi world.

  > However, the conceit is that "in the same repo" means you can
  > "change them together."
In a monorepo you shouldn't be making changes to independent components in a single commit. That's how you end up being forced to roll back your change because you broke someone else's service.

If you're making a backwards-incompatible change to an API then you need to:

1. Make a commit to your library to add the new functionality,

2. Send separate commits for review by other teams to update their projects' code,

3. Wait for them to be approved and merged in, then merge a final cleanup commit.

If your repository is designed to enable a single commit to touch multiple independent projects then it's not a monorepo, it's just a single-project repo with unclear API and ownership boundaries.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#122

Post by ikekkdcjkfke »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

1) You can have several independent projects in a monorepo

2) Private/public/internal modifiers

3) Independent builds/project in a monorepo

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#123

Post by 0x6c6f6c »

yongjik wrote: post #84
spion wrote: post #51

Earlier quoted context omitted.

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Effort to update something is high because there's a lot of code, not because it's in a monorepo. Updating the same code scattered across multiple repositories takes as much work in the best case. More realistically, some copy of the same code will stay unupdated because the cost to track down every repository in the company is too much.

Can definitely feel this pain personally. Need to upgrade tooling across some dozen or so services and we're investigating how to migrate with potentially incompatible upgrades. So just suffer outage while we merge PRs across some 20 repos? The atomic changes of a monorepo are very beneficial in these cases, removing the manual orchestration of GitOps practices segmented across individual services..

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#124

Post by 0x6c6f6c »

quantum_state wrote: post #85

something is seriously wrong if Google needs 2B loc to do its things …

How do you propose you provide the number of services Google has without lots of code? For context, the entirety of the Google suite is in there, and a lot more. I'm even somewhat surprised it's that little with their scale.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#125

Post by safog »

yazaddaruvala wrote: post #22

Having worked at Google and Amazon. Honestly their systems are almost identical. Amazon just creates a monotonically increasing watermark outside the “repo”. Google uses “the repo” to create the monotonically increasing watermark. Otherwise, Google calls it “merge into g3” Amazon calls it “merge into live”. Amazon has the extra vocabulary of VersionSets/Packages/Build files. Google has all the same concepts, but just…

There are no presubmits that prevent breaking changes from "going into live". If some shared infra updates are released, the merge from live breaks for multiple individual teams rather than preventing the code from getting submitted in the first place.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#126

Post by ywoski »

spion wrote: post #51
gresrun wrote: post #44

Earlier quoted context omitted.

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Almost. We had a UI library on Android that was stuck on an alpha version of the library for three or so years after the library had shipped.

Upgrading the library broke many tests across the org, and no one wanted to own going in and getting each team to fix it. Eventually, the library had a v2 release, and people started to care about being able to use it.

Ultimately, they just forked the current release and appended a v2 to the package name.

Not the norm, but it happens. The monorepo works for Google, but I wouldn't recommend it for most organizations; we have a ton of custom tooling and headcount to keep things running smoothly.

From the mobile side, it makes it super easy for us to share code across the 50+ apps we have, manage vulnerabilities quicker, and collaborate easily across teams.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#127

Post by saalweachter »

zelphirkalt wrote: post #90
myko wrote: post #62

Earlier quoted context omitted.

> Sure, but this is unsustainable. Not exactly unsustainable considering Google has been very successful with this approach!

Is this a good comparison? Not everyone is Google-size. In fact, very few businesses are. What is managable for Google or a good practice for Google, might be unsustainable for another business.

I think the lesson to draw from bigorgs isn't what to as a smallorg, but what directions you can grow and what the pitfalls are on those roads.

Any smallorg probably wants a bare monorepo, git or what have you. If you grow to the point that becomes unwieldy, you can either invest in tooling the way Google has, or be prepared to split the repo into library and project repos in a way that makes sense for what your mediumorg has grown into.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#128

Post by taeric »

jmillikin wrote: post #121
taeric wrote: post #116

Earlier quoted context omitted.

In a multi-repo, you don't build source dependencies between projects. You can do this with a mono, as well. However, the conceit is that "in the same repo" means you can "change them together." It is very very tempting that "went out as a single commit" means that it went out fine. Which, just isn't something you see in a multi world.

> However, the conceit is that "in the same repo" means you can > "change them together." In a monorepo you shouldn't be making changes to independent components in a single commit. That's how you end up being forced to roll back your change because you broke someone else's service. If you're making a backwards-incompatible change to an API then you need to: 1. Make a commit to your library to add the new functionali…

This is clearly correct. But even in a multi world, I've seen far more attempts at atomic commits than makes sense.

I'd love for it to be a strawman. But I do keep finding them.

You do get me to question what a mono repo is. I've never seen one that wasn't essentially an attempt at treating a company as a large project. Akin to a modular codebase with a single build. Could be a complicated build, mind you. Still, the goal has always been a full repository build.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#129

Post by skybrian »

yongjik wrote: post #84
spion wrote: post #51

Earlier quoted context omitted.

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Effort to update something is high because there's a lot of code, not because it's in a monorepo. Updating the same code scattered across multiple repositories takes as much work in the best case. More realistically, some copy of the same code will stay unupdated because the cost to track down every repository in the company is too much.

When you say it's "as much work" there's an assumption the code is still used. This was years ago, but when I was doing migrations at Google we sometimes had to deal with abandoned or understaffed and barely maintained code. (Sometimes by deleting it, but it can be unclear whether code by some other team is still useful.)

If you're not responsible for fixing downstream dependencies then you don't need to spend any time figuring that out.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#130

Post by coryrc »

ywoski wrote: post #126
spion wrote: post #51

Earlier quoted context omitted.

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Almost. We had a UI library on Android that was stuck on an alpha version of the library for three or so years after the library had shipped. Upgrading the library broke many tests across the org, and no one wanted to own going in and getting each team to fix it. Eventually, the library had a v2 release, and people started to care about being able to use it. Ultimately, they just forked the current release and append…

> The monorepo works for Google

Does it? Or is it stopping Google from supporting products which only make millions in revenue because of the massive burden of continually updating?

Post reply on HN