Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

271–280 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#271

Post by shados »

AmericanChopper wrote: post #197
shados wrote: post #114

Earlier quoted context omitted.

Both monorepo or "micro repo" end up falling apart at scale without some devops work involved. Either will work if you only have a few dozen projects. Neither will work once you hit 10s of millions of lines of code. But people seem to forget that it wasn't that long ago that git didn't exist, making multiple repos was a pain in the butt. Managing multiple repos locally was hell. Monorepos were the norm. Then as the s…

>Both monorepo or "micro repo" end up falling apart at scale without some devops work involved Wouldn’t any project fall apart without devops work?

yeah, that's my point.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#272

Post by pcwalton »

dekhn wrote: post #79
pcwalton wrote: post #32

Earlier quoted context omitted.

There's also the fact that monorepos have issues when you don't have one organization responsible for all the code. The Linux kernel and NetHack don't live in the same repository for good reason.

I dunno, the BSD distribution included a wide gamut of games along with the kernel source in the same tree. In fact, NetHack is derived from Hack which itself is derived from Rogue, which was distributed within BSD. And BSD represented a cross-organization responsibility (see the history of AT&T and BSD).

Fine, replace NetHack with Quake 3. :)

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#273

Post by joshuamorton »

flukus wrote: post #167
joshuamorton wrote: post #138

Earlier quoted context omitted.

Yeah that second thing doesn't exist. That first thing doesn't really exist the way you conceptualize either, I don't think.

Can anyone articulate how it does work and where I'm going wrong then? The conversations feeling pretty one sided here. You said the first thing doesn't exist? Do you not have local changes or are these changes not shared with the test/build server? Having a set of patches, code changes, whatever sounds like a branch to me, are you being too literal with the word branch? For the second part what doesn't exist? Do you…

I can try. I think people are averse to doing this because it can sort of require a deep dive into how Piper and Citc work, and the linked article does a good job of explaining that, and beyond what the article says, its not clear what you can discuss.

[Everything I'm about to explain is for the average user's workflow, like others have mentioned, "real" branches do exist, but most engineers will never use them, and my current workflow works differently than what I'm explaining, but I used to do it this way.]

Piper generally speaking doesn't have the concept of commits or "sets of patches". You have clients. A client trunk@time + some local changes. You could maybe call this a branch, but you can't stack multiple commits[1], so its a branch of length exactly one. It can only be merged back into trunk. Then you delete the client and start a new one. You can patch changes from one client into another, but this isn't generally done or super useful because again, you can't stack changes.

A given client has an owner, and the owner has write access. Everyone else has read access.

So to answer your questions:

>Do you not have local changes or are these changes not shared with the test/build server?

There are local (sort of) changes. And you can test/build them, but they lack many of the concept one would expect of a branch, so I'm not sure that's a good name for them.

>Do you not make changes that breaks other peoples code?

Sure you do. But you're responsible for fixing it (as I said elsewhere).

>Do you not get them to help fix it?

Yeah, but normally this is done by having them review the change, or talking in person. There's nothing like multiple commits by multiple people which are then squashed and merged.

>Can you not share your work in progress changes with others?

Sure, but they can't edit them.

>Can you goes collaborate on changes at all?

Kind of, but not with multiple authors.

[1]: There's a hack that allows chained CLs, but its a hack, a leaky abstraction, and still doesn't provide multiple authors squashing and merging.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#274

Post by vkou »

flukus wrote: post #219
vkou wrote: post #209

Earlier quoted context omitted.

You can have versioned packages inside a mono-repository, too, though. /common_libs/foo_lib_v1.13/, /common_libs/foo_lib_v1.14/, etc.

By that point your creating micro-repositories in your mono-repository and getting the worst of both worlds.

No, you're getting the best of both worlds, because it's incredibly clear to the infrastructure maintainers what version everyone's on, whether or not the old version can be safely deprecated, who is responsible for deprecating it, etc.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#275

Post by Conan_Kudo »

malkia wrote: post #178
lpghatguy wrote: post #165

Earlier quoted context omitted.

Most popular Git frontends (GitHub and GitLab too, I believe) let you link to commits with just the first 5-6 characters of the hash. I don't think that's much different to remember than a Perfore CL number.

To me the issue is when mentally trying to work with these numbers, P4 & G4's numbers increment, so I can tell which one came before the other - I can't do this with hashes. I'm sure I can get used to the other way, but this cannot easily be ignored.

Mercurial gives you those incrementing numbers through its revlog, and they map to revisions, so you get that facility there.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#276

Post by joshuamorton »

Too wrote: post #221
jgibson wrote: post #194

Is it just me, or are a lot of people here conflating source control management and dependency management? The two don't have to be combined. For example, if you have Python Project X that depends on Python Project Y, you can either have them A) in different scm repos, with a requirements.txt link to a server that hosts the wheel artifact, B) have them in the same repo and refer to each other from source, or C) have…

This is my biggest gripe in discussions like this as well, dependency management and source control are two completely different things. It should be convenient to use one to find the other but they should not necessarily be 1-1 coupled together with each other. 1. A single repo should be able to produce multiple artifacts. 2. It should be possible to use multiple repos to produce one artifact. 3. It should be possib…

Number three I disagree with. Bisection depends on build (and test) always working on trunk.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#277

Post by rkangel »

thomaslee wrote: post #234
edejong wrote: post #222

Earlier quoted context omitted.

The key here is reverse dependency management. “If I change X, what would influence this change?”. This can be achieved with single repo better than multi-repo due to the completeness of the (dependency) graph.

Exactly this. Or at least it's a way this can be achieved, assuming solid testing & some tooling in the mix. For folks unfamiliar with it, the issue is something like: 1. You find a bug in a library A. 2. Libraries B, C and D depend on A. 3. B, C and D in turn are used by various applications. How do you fix a bug in A? Well, "normal" workflow would be something like: fix the bug in A, submit a PR, wait for a CI buil…

There's a subtler, and potentially more important thing that can crop up with your scenario:

Library A realises that its interface could be improved, but it would not be backwards incompatible. In the best case scenario, with semver, there is a cost to this change. Users have to bump versions and rewrite code, maybe the maintainer of Library A has to keep 2 versions of a function to ease the pain for users. It may just be that B, C and D trust A less because the interface keeps changing. All this can mean an unconscious pressure to not change and improve interfaces, and adds pain when they do.

Doing it in a monorepo can mean that the developers of A can just go around and fix all the calls if they want to make the change, allowing for greater freedom to fix issues with interfaces between modules. And that is really important in large complex systems with interdependent pieces.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#278

Post by nwlieb »

robaato wrote: post #182
makecheck wrote: post #120

This is clearly detrimental to external projects such as Go packaging, since their own developers will never be looking at dependency problems in the same way as outside groups. Monorepo also bugs me because there will always be some external package you need, and invariably it’s almost impossible to integrate due to years of colleagues making internal-only things assume everything imaginable about the structure and…

What about Android and 800-1,000 git repos?! Have seen the pain trying to manage that across larger teams (e.g. thousands of devs) - and no the "repo" tool is not sufficient.

I'm very curious, what pain did you see with the repo.py tool?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#279

Post by wirrbel »

marmaduke wrote: post #169
wirrbel wrote: post #153

Earlier quoted context omitted.

Same line of thinking, just different conclusions. I feel terrible for anyone trying to run a company with open-source style independent repos. On a popular github project, you have MANY potential contributors that will tell you if a PR, or a release candidate break API compatibility, etc. There are thousands of hours in open source dedicated to fixing integration issues due to the (unavoidable) poly-repo situation.…

working as dev with academic teams, I usually use many repos for "damage control" as git-ignorant scientists will dump irrelevant files into a repo. with that in mind, is monorepo is a universally good approach or is more dependent on good behavior of team members than polyrepo?

A monorepo requires a good Continuous Integration infrastructure if it is supposed to work. Unless those small repos are will be unit tested, you will not benefit from a monorepo.

Suppose for your projects you have a utility library `lib_a`, in a polyrepo situation, your projects will use it in probably different versions, which means you have coordination effort necessary to get everyone on the latest release. The monorepo would enable the developers of `lib_a` to get feedback from the downstream test suites directly on whether the changes they perform are breaking user code, so they can up front introduce their changes less intrusive. They can however also roll out security-relevant changes much more easily. The monorepo will make the projects more homogeneous, which facilitates integration and operations (there are exceptions of course).

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#280

Post by 3minus1 »

013a wrote: post #173
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

I call this "Google Imposter Syndrome". Because Google (insert Facebook, Apple, Amazon, etc) has success with Monorepos (insert gRPC, Go, Kubernetes, React/Native, etc), it must be a great idea, we should do it. You see this everywhere . Also known as an Appeal to Authority. My personal opinion: very few companies will hit a point where sheer volume of code or code changes makes a monorepo unwieldy. Code volume is a…

also known as "cargo culting"
Post reply on HN