Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

171–180 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#171

Post by oikawa_tooru_ »

gresrun wrote: post #44
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

Hi, unrelated to this, but since you are working at Google, were there actually "code red" meetings at Google concerning chatgpt?

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#172

Post by robertlagrant »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> 2. It encourages poor API contracts because it lets anyone import any code in any service arbitrarily. Shared functionality should be exposed as a standalone library with a clear, well-defined interface boundary. There are entire packaging ecosystems like npmjs and pypi for exactly this purpose.

I don't believe this is true, except in the short term. Unless the writing party is guaranteeing you forward compatibility, your consuming code will break when you update.

This is (almost) the only reason API contracts are worth having; the reason doesn't go away just because you can technically see all the code.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#173

Post by robertlagrant »

gorgoiler wrote: post #150

Imagine you have two teams in one monorepo and requirements.txt has pinned numpy at 1.22. One team wants to upgrade to 1.24 but the upgrade breaks the other team’s code as it was dependent on an emergent property* in the older version of numpy. How would you handle this situation as an IC? As a manager of one of the teams? As a skip-level manager of both teams? As a budding IC on the team that wants the upgrade, you…

I haven't come across the concept before that the monorepo has to have one set of dependencies. Why not just have different dependencies in different projects' folders?

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#174

Post by rfoo »

dx034 wrote: post #162
skybrian wrote: post #129

Earlier quoted context omitted.

When you say it's "as much work" there's an assumption the code is still used. This was years ago, but when I was doing migrations at Google we sometimes had to deal with abandoned or understaffed and barely maintained code. (Sometimes by deleting it, but it can be unclear whether code by some other team is still useful.) If you're not responsible for fixing downstream dependencies then you don't need to spend any ti…

Sounds great to me because you are forced to delete code that's not in use anymore. Without the monorepo, that code would still be there with old libraries that are potentially insecure. Deleting code that is not being used anymore happens way too rarely in my opinion.

The downside is if a product no longer have maintainers you are now encouraged to shut it down, even if it still works and it doesn't cost much to run.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#175

Post by rfoo »

spion wrote: post #168
yongjik wrote: post #158

Earlier quoted context omitted.

The flip side is that services with an immediate need will get upgraded, and others won't, and six months later you will be saying "Why am I still seeing this bug in production, I already fixed it three times!" Of course, the problem can be mitigated by a disciplined team that understands the importance of everybody being on the same page on which version of each library one should use. On the other hand, such a team…

I don't think that's quite true. In my experience multi-repos have the edge here. If you have one key dependency update with a feature you need, but you need substantial code updates and 80 services depend on it, that may be impossible to pull off no matter what. Comparatively, upgrading one by one may not be easy, but at least its possible. The importance of everyone being on the same page with dependencies might ju…

There are languages / runtimes where there could not be two different versions of the same thing in one binary (and they eagerly fail at build time / immediately crash upon run). That is not the case for JavaScript, Rust, etc. But it is the case for C++, Java, Go, Python and more.

Everyone claims different needs if they can. Nothing could be linked together anymore if you just let everyone use whatever they want.

Or maybe people start to try to workaround this by ... reinventing the wheels (and effectively forks and vendoring) to reduce their dependency graph.

There is a genuine need for single instance of every third party dependencies. It is not unique to monorepos. Monorepo (with corresponding batch change tooling) just make this feasible, so you don't hear about this concept for manyrepos, and mentally bind it to monorepo.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#176

Post by rfoo »

mook wrote: post #142
ywoski wrote: post #126

Earlier quoted context omitted.

Almost. We had a UI library on Android that was stuck on an alpha version of the library for three or so years after the library had shipped. Upgrading the library broke many tests across the org, and no one wanted to own going in and getting each team to fix it. Eventually, the library had a v2 release, and people started to care about being able to use it. Ultimately, they just forked the current release and append…

> Ultimately, they just forked the current release and appended a v2 to the package name. Hmm, does that explain the golang module versioning requirement where v2 must have a different name?

Yes.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#177

Post by zelphirkalt »

saalweachter wrote: post #127
zelphirkalt wrote: post #90

Earlier quoted context omitted.

Is this a good comparison? Not everyone is Google-size. In fact, very few businesses are. What is managable for Google or a good practice for Google, might be unsustainable for another business.

I think the lesson to draw from bigorgs isn't what to as a smallorg, but what directions you can grow and what the pitfalls are on those roads. Any smallorg probably wants a bare monorepo, git or what have you. If you grow to the point that becomes unwieldy, you can either invest in tooling the way Google has, or be prepared to split the repo into library and project repos in a way that makes sense for what your medi…

What is a "bare monorepo" in contrast to simply a "monorepo"?

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#178

Post by jen20 »

fouronnes3 wrote: post #157
gorgoiler wrote: post #150

Imagine you have two teams in one monorepo and requirements.txt has pinned numpy at 1.22. One team wants to upgrade to 1.24 but the upgrade breaks the other team’s code as it was dependent on an emergent property* in the older version of numpy. How would you handle this situation as an IC? As a manager of one of the teams? As a skip-level manager of both teams? As a budding IC on the team that wants the upgrade, you…

While I found your comment insightful and sadly very accurate, it's fundamentally a human problem, not a technical problem. So I don't think the solution to it should be technical like "don't use a monorepo and those problems will go away!", but rather organisational in nature.

Specifically, the companies that encourage (or permit) these kinds of problems and people to prevail should fail.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#179

Post by humanrebar »

zelphirkalt wrote: post #169
safog wrote: post #125

Earlier quoted context omitted.

There are no presubmits that prevent breaking changes from "going into live". If some shared infra updates are released, the merge from live breaks for multiple individual teams rather than preventing the code from getting submitted in the first place.

With an appropriately configured CI pipeline, submitted / pushed code does not go live anyway, unless all tests and other checks pass. Unless a test case is missing, which can happen in a mono repo just as well, the code is always checked for the defect.

It's impossible to test for every kind of regression. Concurrency and performance bugs are notoriously problematic. At the scales of large codebases, you can have very thorough tests, but they need to be reasonably fast and behave the same way every time they run.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#180

Post by pjmlp »

rfoo wrote: post #175
spion wrote: post #168

Earlier quoted context omitted.

I don't think that's quite true. In my experience multi-repos have the edge here. If you have one key dependency update with a feature you need, but you need substantial code updates and 80 services depend on it, that may be impossible to pull off no matter what. Comparatively, upgrading one by one may not be easy, but at least its possible. The importance of everyone being on the same page with dependencies might ju…

There are languages / runtimes where there could not be two different versions of the same thing in one binary (and they eagerly fail at build time / immediately crash upon run). That is not the case for JavaScript, Rust, etc. But it is the case for C++, Java, Go, Python and more. Everyone claims different needs if they can. Nothing could be linked together anymore if you just let everyone use whatever they want. Or…

> But it is the case for C++, Java, Go, Python and more.

It certainly isn't for Java, hence why multiple classloaders exist.

For C and C++ it depends on the OS, on Windows (AIX, and similar OSes) this isn't an issue thanks to how symbol visibility works.

Two different libraries are free to have whatever versions they feel like.

Post reply on HN