Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

21–30 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#21
> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world.

> Access to the whole codebase encourages extensive code sharing and reuse [...]

Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#22
Having worked at Google and Amazon.

Honestly their systems are almost identical. Amazon just creates a monotonically increasing watermark outside the “repo”. Google uses “the repo” to create the monotonically increasing watermark.

Otherwise, Google calls it “merge into g3” Amazon calls it “merge into live”.

Amazon has the extra vocabulary of VersionSets/Packages/Build files. Google has all the same concepts, but just calls them Dependencies/Folders/Build files.

Amazon’s workflows are “git-like”, Google is migrating to “git-like” workflows (but has a lot of unnecessary vocabulary around getting there - Piper/Fig/Workspace/etc).

I really can’t tell if the specific difference between “mono-repo” or “multi-repo” makes much practical difference to the devs working on either system.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#23
post #10

Monorepos are great... but only if you can invest in the tooling scale to handle them, and most companies can't invest in that like Google can. Hyrum Wright class tooling experts don't grow on trees. A good article to reference when this topic gets raised: http://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-yo...

With the advent of great CI tooling like GitHub actions, simple monorepos are becoming more and more viable and in fact even recommendable.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#24

> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world. > Access to the whole codebase encourages extensive code sharing and reuse [...] Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.

Most source code just isn't that interesting or sensitive.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#25

> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world. > Access to the whole codebase encourages extensive code sharing and reuse [...] Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.

If you had every line of code that Google wrote, what would you do with it?

But I found this discussion on HN.

https://news.ycombinator.com/item?id=11790438

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#26
post #6

I really wish they would make this tech available via gcloud. Seems like it would be very popular and a great way to attract other gcloud business away from MS/GitHub which scales horribly.

Beating Git's network effects sounds extremely difficult, especially since very few users of Git run into serious problems scaling it.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#27

> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world. > Access to the whole codebase encourages extensive code sharing and reuse [...] Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.

If you had every line of code that Google wrote, what would you do with it? But I found this discussion on HN. https://news.ycombinator.com/item?id=11790438

Well, if you had the search ranking algorithms or the bot-detection algorithms or anything inherently adversarial like that, then you could do all kinds of nefarious things. But that stuff's locked down more tightly. Likewise with a few ultra-hard-tech things where the implementation's a major competitive edge.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#28
post #8
post #6

I really wish they would make this tech available via gcloud. Seems like it would be very popular and a great way to attract other gcloud business away from MS/GitHub which scales horribly.

They tried that by making a bit available with a remote cloud builder for Bazel. It failed for some reason and they pulled it. I think building something that scales for one big repo is just a completely different problem than making it scale for a lot of small repos.

Bazel is not failing in the open source world though

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#29

> Google’s codebase is shared by more [...] than 25,000 Google software develop- ers from dozens of offices in countries around the world. > Access to the whole codebase encourages extensive code sharing and reuse [...] Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.

If you had every line of code that Google wrote, what would you do with it? But I found this discussion on HN. https://news.ycombinator.com/item?id=11790438

I imagine looking for vulnerable areas of the code might be something people would be interested in doing. Maybe start with login or billing or something. You could also look at recent activity to spot new, unannounced projects. You could use blame to find who wrote what and target them for anything from job offers to social engineering attacks.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#30

No wonder noone at Google can't ship everything if they constantly have to stop development of their feature so they can do mandatory upgrades of their dependencies...

Most of that work is done by the owners of the dependencies, rather than the dependents.

This is sometimes a problem for open source dependencies, though, as there isn't always anyone whose job it is to keep them up to date. Some amount of NIH syndrome is because reinventing the wheel can be less work than integrating an existing wheel that was designed for a different vehicle with different specs.

Post reply on HN