Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

81–90 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#81

Post by tfsh »

charcircuit wrote: post #64
gardenhedge wrote: post #54

I've never experienced a monorepo like Googles. How does it work? Are Chrome and Gmail in the same repo? I assume they're built separately and pushing code to one doesn't affect the other.

No, Chrome and Gmail are in different monorepos. >How does it work? Different projects are in different folders instead of different repos. >I assume they're built separately and pushing code to one doesn't affect the other. Yes, building or testing something only builds its dependencies.

For GP, note Chrome is a special case because it's an open source-first project so it is not in the same repo as Gmail.

However products are in the same repo such a gmail, youtube, search (frontend, mobile, server, infra, etc), photos, maps, play, translate and literally thousands of other internal and external products and projects.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#82

Post by [deleted] »

gresrun wrote: post #44
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

[deleted]

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#83

Post by bananapub »

Macha wrote: post #66
Lexarius wrote: post #38

Earlier quoted context omitted.

Despite almost everything being in one big repo, it has silos. Not everyone has read access to everything. Some code, like the important bits of Search, is only available on a need-to-know basis.

So what happens if search adopts your internal library and your update to it breaks search? Do you need to get someone from the search team to go investigate? How is that prioritised?

it's search's fault for locking down their code and so search gets to fix it.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#84

Post by yongjik »

spion wrote: post #51
gresrun wrote: post #44

Earlier quoted context omitted.

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Effort to update something is high because there's a lot of code, not because it's in a monorepo. Updating the same code scattered across multiple repositories takes as much work in the best case. More realistically, some copy of the same code will stay unupdated because the cost to track down every repository in the company is too much.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#86

Post by blandflakes »

lopkeny12ko wrote: post #57
brianwawok wrote: post #52

Earlier quoted context omitted.

Not updating old libraries is how you end up getting known security vulns years after they are patched.

You should ask your colleagues who work in critical industries like banking and healthcare how much of their software stack depends on things that haven't been patched in more than 20 years ;)

Both industries that are notorious for poor security hygiene, so I'm not sure this is the coup you were looking for.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#87

Post by fatneckbeard »

scarface74 wrote: post #25
KolmogorovComp wrote: post #21

> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world. > Access to the whole codebase encourages extensive code sharing and reuse [...] Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.

If you had every line of code that Google wrote, what would you do with it? But I found this discussion on HN. https://news.ycombinator.com/item?id=11790438

id make an art piece where each line of source code was printed 1mm high on the walls of a room. . . . .. . and then..... it would show the program counter "live" on the wall as code was executed. like it would shine a light on the line of code, like a realtime debugger.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#88

Post by jcranmer »

lopkeny12ko wrote: post #59
jmillikin wrote: post #46

Earlier quoted context omitted.

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

> There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd hav…

So why is this a problem for a monorepo but not the multi-repo? It seems to me that the major difference is that in a multi-repo, you'd be more likely to be oblivious to the multitude of dependency issues than you are in the monorepo... and to be honest, that actually sounds like a bad thing to me, because it means you're sweeping legitimate issues underneath the rug.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#90

Post by zelphirkalt »

myko wrote: post #62
lopkeny12ko wrote: post #59

Earlier quoted context omitted.

> There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd hav…

> Sure, but this is unsustainable. Not exactly unsustainable considering Google has been very successful with this approach!

Is this a good comparison? Not everyone is Google-size. In fact, very few businesses are. What is managable for Google or a good practice for Google, might be unsustainable for another business.
Post reply on HN