Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

Post reply on HN
Print view View original

91–100 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#91

Post by zelphirkalt »

jmillikin wrote: post #46
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

I guess the question then becomes: Is it worth all the extra tooling required to manage a monorepo properly?

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#92

Post by ASinclair »

aleksiy123 wrote: post #49
ghosty141 wrote: post #45

Earlier quoted context omitted.

How do you deal with wanting to see the history, graph etc of just one sub-project? Does the tooling handle this?

I believe everything is tracked at the folder/file level and not a project level. I'm not sure there even is a concept of a project. But maybe someone can correct me.

There is a concept of a project. Though viewing change history is more organized around packages and files.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#93

Post by ASinclair »

jmillikin wrote: post #46
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

There’s a policy against multiple versions of third party dependencies. Though there is a mechanism for exceptions.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#94

Post by faizshah »

yazaddaruvala wrote: post #22

Having worked at Google and Amazon. Honestly their systems are almost identical. Amazon just creates a monotonically increasing watermark outside the “repo”. Google uses “the repo” to create the monotonically increasing watermark. Otherwise, Google calls it “merge into g3” Amazon calls it “merge into live”. Amazon has the extra vocabulary of VersionSets/Packages/Build files. Google has all the same concepts, but just…

I haven’t worked at google but I think there is one other difference. At amazon teams “merge from live” and have control of their own service’s CD pipeline. They might manually release the merged changes or have full integ test coverage. The Amazon workflow offers more flexibility to teams (whether or not that might be desirable).

Not sure how deployments and CD work at google but I think the picture is different at google for unit tests, integ tests etc. Amazon teams have more control over their own codebase and development practices whereas, based on what I know, google has standardized many parts of their development process.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#95

Post by throwaway2037 »

jmillikin wrote: post #46
lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

> The single version dependencies are asinine. We are migrating to > a monorepo at work, and someone bumped the version of an open > source JS package that introduced a regression. There's no requirement to have single versions of dependencies in a monorepo. Google allows[0] multiple versions of third-party dependencies such as jQuery or MySQL, and internal code is expected to specify which version it depends on. > I…

I never worked at Google, but this post sums up everything I had to say about the matter. GP has a sh-tty monorepo experience at one company and decides to make a statement about another company where they never worked (so I presume). HN absurdism as its best!

I second your point about monorepo versus ball of mud. They are so different. And managing all of this is about social/culture, less science-y. If you don't have good culture around maintenance, well then, yeah, duh, it will fall apart pretty quickly. It sounds like Google spends crazy money to develop tools to enforce the culture. Hats off.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#96

Post by Karellen »

bananapub wrote: post #80
Karellen wrote: post #56

> The Google codebase includes approximately one billion files and has a history of approximately 35 million commits spanning Google’s entire 18-year existence. Wait, that's an average of nearly 30 new files per commit. Not 30 files changed per commit, but whatever changes are happening to existing files, plus 30 brand new files. For every single commit. Although... > The total number of files also includes source fi…

you're misunderstanding a bunch of things. > The total number of files also includes source files copied into release branches I guess you haven't used Perforce or similar. a branch is a sparse copy of just the changed files/directories. they are not used very much. > files that are deleted at the latest revision so it means "one billion files have existed in the history repo, some are currently deleted". > I don't t…

> you're misunderstanding a bunch of things.

It certainly feels that way :-)

> > The total number of files also includes source files copied into release branches

> I guess you haven't used Perforce or similar. a branch is a sparse copy of just the changed files/directories.

Still not sure I see the distinction. Surely "sparse" or "not sparse" is an implementation detail. If I create a new branch in git, the files that are unchanged from its parent branch share the same storage, but the files that have changed use their own storage.

> so it means "one billion files have existed in the history repo, some are currently deleted".

I guess I'm struggling to understand what the point of this metric is? I get why "Total number of commits", "Total storage size of repo in GB/TB/PB", "Number of files in current head/main/trunk", or even "total number of distinct file revisions in repo history", could be useful metrics.

But why "number of files (including ones that have been deleted)"? What can we do with this number?

> hope that clears things up.

It's helping. Thanks.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#98

Post by throwaway2037 »

lopkeny12ko wrote: post #57
brianwawok wrote: post #52

Earlier quoted context omitted.

Not updating old libraries is how you end up getting known security vulns years after they are patched.

You should ask your colleagues who work in critical industries like banking and healthcare how much of their software stack depends on things that haven't been patched in more than 20 years ;)

    "critical industries like banking and healthcare".
What a red herring. This comment reads like ChatGPT was trained on Reddit forums. 99% of the software in those industries runs "inside the moat" where security doesn't matter. I am still running log4j from 10 years ago in lots of my stack, and it is the swiss cheese of software security! Who cares! It works! I'm inside the moat! If people want to do dumb black hat stuff, they get fired. Problem solved.

Also what does "banking" mean anyway? That comment is so generic as to be meaningless. If you are talking about Internet-facing retail banks in 2023, most are very serious about security... because regulations, and giants fines when they get it wrong. And if the fines aren't large enough in your country, tell your democratically elected officials to 10x the fines. It will change industry behaviour instantly -- see US investment banks' risk taking after the Vocker Rule/Dodd-Frank regulations.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#99

Post by summerlight »

lopkeny12ko wrote: post #35

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

While 2nd and 3rd points are not really something unique to monorepo, the first point is actually valid. This is why monorepo usually should be packaged with bunch of other development practices, especially comprehensive tests combined with presubmit hook.

IMO, it's more of a development paradigm rather than a mere technology. You cannot simply use monorepo in isolation since its trade-off is strongly coupled with many other tooling and workflow. Because of this reason, I usually don't recommend migration toward monorepo unless there's strong organizational level support.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#100

Post by throwaway2037 »

patrick451 wrote: post #20
zdw wrote: post #10

Monorepos are great... but only if you can invest in the tooling scale to handle them, and most companies can't invest in that like Google can. Hyrum Wright class tooling experts don't grow on trees. A good article to reference when this topic gets raised: http://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-yo...

You don't need google scale tooling to work with a mono repo until you are actually at google scale. Gluing together a bunch of separate repos isn't exactly free either. See, for example, the complicated disaster Amazon has with brazil. In the limit, there are only two options: 1. All code lives one repo 2. Every function/class/entity lives in its own repo with a third state in between 3. You accept code duplication…

I don't understand why this was downvoted. Your list of three states is important to the debate. I never saw it that way. Another, more hostile way to put it: "What is a better or worse alternative and why?" Pretty much everything fits into one of those three states -- with warts.
Post reply on HN