Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

131–140 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#131

Post by djur »

nicodjimenez wrote: post #90

I have slight experience with both monorepos and smaller repos and I think they can both work. The advantage of smaller repos is that it forces different components to expose well designed API's. Bigger repos make sense for products and embedded software, smaller repos make sense for platforms build up of small services communicating on the internet.

Smaller repos force different components to expose APIs, but I don't think it forces or even encourages the APIs to be well designed. In some cases, having work spread across multiple repos can impede iterative development, meaning that you risk half-assed or, uh, two-and-a-half-assed implementations.

Also, when someone's asking for review for a change that encompasses, say, a change to a service, a change to a client library for that service, and a change to 2-3 other services that use that client library, I know that I cringe a little when suggesting a change, knowing that to implement it is going to require a commit on all of these different repos, waiting for CI to run on each one, etc. I try to only use that impulse to counter the urge to bikeshed, but the temptation is there.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#132

Post by shados »

fcarraldo wrote: post #123
shados wrote: post #114

Earlier quoted context omitted.

Both monorepo or "micro repo" end up falling apart at scale without some devops work involved. Either will work if you only have a few dozen projects. Neither will work once you hit 10s of millions of lines of code. But people seem to forget that it wasn't that long ago that git didn't exist, making multiple repos was a pain in the butt. Managing multiple repos locally was hell. Monorepos were the norm. Then as the s…

People also seem to forget that "Monorepo" or (many) "Microrepos" is not a binary choice. You can have both tiny repositories which do a single thing and large repositories that consist of many projects. It's totally cool to have both, assuming your team can be trusted to make the appropriate choices as they create new projects. > And then people forget how to make distributed repos work and claim things like "omg I…

> Is this a solved problem

I don't mean that it's magical, just that it's not particularly sorcery. Instead of making a breaking change, add new method, deprecate old method. Update projects, then get rid of old deprecated method. Because they're distinct you can do this one by one so some project can reap the benefits without having to wait until all the problems are solved.

Some people in this thread act like its freagin impossible. Avoiding breaking changes in APIs or proper deprecation strategies is an art everyone developing software should know: sooner or later they'll have to contribute to an open source project or have to make a more complicated breaking change or SOMETHING and will have to deal with it. Even if they use a monorepo. And when it happens you don't want it to be the first time anyone deals with it.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#133

Post by ebikelaw »

flukus wrote: post #126
joshuamorton wrote: post #106

Earlier quoted context omitted.

Your last paragraph doesn't sound like anything at Google. Most engineers will never use branches at all, and even fewer will use branches that merge into trunk (instead of away from it).

There is a set of code changes locally and those changes are bundled off to the test server to run the full test suite? That's a branch. Now let's say I break a project sharing this code and because I'm not an expert in all 2 billion LoC and 3000 projects google is running I need to enlist some help in fixing what I broke. Presumably there is a way for the developers on that downstream project to pull in my change se…

Your mental model of how this works within Google is completely foreign to me. I think you've made an unfounded assumption somewhere.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#134

Post by YokoZar »

u801e wrote: post #122
crazygringo wrote: post #113

Earlier quoted context omitted.

> But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage. Nope. In a monorepo (like at Google), you're responsible for not breaking anyone else's code, as evidenced by their tests still passing. So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first plac…

> So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push. Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?

> Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?

Eventually you hit a point where you need systems to run the tests for you. Making this work is part of the investment in infrastructure and tooling you need to do as a big serious company.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#135

Post by hobls »

rco8786 wrote: post #61
hobls wrote: post #49

Earlier quoted context omitted.

There’s an important distinction between lots of code and lots of projects. I agree; if you have a ton of code you’d better invest in tooling. But if you just have several normally sized projects, a monorepo can make your life much more difficult than simply using several repos.

Sure. You could also replace the term “monorepo” with “separate repos” and your statement would be just as valid. Either way you go has pros and cons.

Agreed. I think the thing is that GitHub basically supports the separate repo approach fairly well out of the box. Using only a single repo requires more thought around your strategy, especially if you have multiple teams.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#136

Post by Tloewald »

adamrt wrote: post #69
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

My previous had a monorepo for the website and backend (but not the mobile apps) which was insane to work with (as a coder I had a dedicated 128 core box to work on, some engineers had more than one, less intense engineers shared one) and a substantial amount of my time was spent just finding code. I guess most engineers just end up working in some nook and so that searching code constantly becomes less of an issue (it never did for me) but the code / debug cycle was dreadful.

I should add that a huge amount was invested in tooling. We had an in-house IDE with debug tools that could step through serverside code. We had a highly optimized code search tool. We had modified a major version control system so it could handle our codebase. (Indeed we picked our version control system because we needed to fork it and the other major version control system was less amenable to our PRs.)

My current job we have a micro service architecture and lots of small, focused repos. Each repo is self-documented. Anyone can checkout and build anything. We don’t need obscene dev servers. We have not hugely invested in tools or workflow.

Client apps are unavoidably larger repos than the services apps.

Based on my personal experience, I think monorepos are nuts.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#137

Post by summerlight »

u801e wrote: post #122
crazygringo wrote: post #113

Earlier quoted context omitted.

> But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage. Nope. In a monorepo (like at Google), you're responsible for not breaking anyone else's code, as evidenced by their tests still passing. So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first plac…

> So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push. Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?

Google has a CI infrastructure which runs most of the affected tests for each commit (which they call "CL") on thousand of machines in parallel. Though even for Google, running the entire test suite every time is prohibitively expensive so they have a way to merge and run multiple CLs in a single batch run every 3 hours, which is useful for testing a CL that may affect hundreds of thousands of build/test targets. If you're interested, this paper may give you an idea how Google is doing test.

https://static.googleusercontent.com/media/research.google.c...

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#138

Post by joshuamorton »

flukus wrote: post #126
joshuamorton wrote: post #106

Earlier quoted context omitted.

Your last paragraph doesn't sound like anything at Google. Most engineers will never use branches at all, and even fewer will use branches that merge into trunk (instead of away from it).

There is a set of code changes locally and those changes are bundled off to the test server to run the full test suite? That's a branch. Now let's say I break a project sharing this code and because I'm not an expert in all 2 billion LoC and 3000 projects google is running I need to enlist some help in fixing what I broke. Presumably there is a way for the developers on that downstream project to pull in my change se…

Yeah that second thing doesn't exist. That first thing doesn't really exist the way you conceptualize either, I don't think.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#140

Post by joatmon-snoo »

flukus wrote: post #126
joshuamorton wrote: post #106

Earlier quoted context omitted.

Your last paragraph doesn't sound like anything at Google. Most engineers will never use branches at all, and even fewer will use branches that merge into trunk (instead of away from it).

There is a set of code changes locally and those changes are bundled off to the test server to run the full test suite? That's a branch. Now let's say I break a project sharing this code and because I'm not an expert in all 2 billion LoC and 3000 projects google is running I need to enlist some help in fixing what I broke. Presumably there is a way for the developers on that downstream project to pull in my change se…

Google internally has many concepts of branches. What you describe is not any of them.
Post reply on HN