Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

121–130 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#121

Post by shados »

tiglionabbit wrote: post #84
flukus wrote: post #58

Earlier quoted context omitted.

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

What you're proposing goes a step beyond multiple repos and into package versioning. That is one viable workflow: Make a change to the common code and publish it as a new package version while allowing all existing code to continue to use the old package. Then, migrate other projects to the newer version of the dependency one by one. Allowing multiple versions of the same code to exist in production at once adds comp…

> Also, if you're doing this with code that is ultimately webpacked to run in a web browser and you don't pay attention to the full tree of dependencies you're working with, there's a chance you end up loading two versions of the same library into a single web page, increasing the page weight and possibly causing incompatibilities in event handling.

You probably should have a way to visualize bundle size increases in PRs easily, so that this becomes obvious. Alternatively, some package managers like Yarn let you flatten the dependency tree, forcing you to pick one version of everything. Even with a monorepo, since you'll likely be using 3rd party dependencies, it's always an interesting exercise because of how hard NPM makes this: getting to a point where you only have 1 version of every 3rd party package can be very, very hard as some combinations of libs are mutually exclusive.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#122

Post by u801e »

crazygringo wrote: post #113
whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

> But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage. Nope. In a monorepo (like at Google), you're responsible for not breaking anyone else's code, as evidenced by their tests still passing. So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first plac…

> So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push.

Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#123

Post by fcarraldo »

shados wrote: post #114
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

Both monorepo or "micro repo" end up falling apart at scale without some devops work involved. Either will work if you only have a few dozen projects. Neither will work once you hit 10s of millions of lines of code. But people seem to forget that it wasn't that long ago that git didn't exist, making multiple repos was a pain in the butt. Managing multiple repos locally was hell. Monorepos were the norm. Then as the s…

People also seem to forget that "Monorepo" or (many) "Microrepos" is not a binary choice.

You can have both tiny repositories which do a single thing and large repositories that consist of many projects. It's totally cool to have both, assuming your team can be trusted to make the appropriate choices as they create new projects.

> And then people forget how to make distributed repos work and claim things like "omg I have to make 1 PR per repo when making breaking changes!", as if it was a big deal or it wasn't a solved problem.

Is this a solved problem? I typically do make one PR per repo to resolve breaking changes, though it's certainly not a big deal. Still, if there's an easier way, I'd love to hear about it!

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#124

Post by shados »

brazzledazzle wrote: post #72
flukus wrote: post #58

Earlier quoted context omitted.

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

I don't have a horse in this race but if you have, say, a security issue and that needs to propagate downstream where does your responsibility end in that situation? Do you try to track down the dependencies and open issues in their trackers? Or maybe a more common problem is a change to a library that consumes an API that's changing so updating the library has a drop dead date.

As a data point, we put the dependency graphs in a database at build time. When we have an emergency and need to push a library update to thousands of repos, we make the change, then trigger builds for all the dependents. We don't auto deploy (too risky), but we use the data we have to start nagging the owner of all the repositories to tell them they have to deploy asap. Since all projects are very small, they build and deploy very, very quickly (a few minutes at most for the big ones).

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#125

Post by djur »

haglin wrote: post #86

Google's handling of their source code makes me wanna work there. I don't like distributed version control systems with hundreds of repositories spread out. It makes management more complicated. I understand this is a minority view, but that is my experience. It was easier to work in a single Perforce repository than hundreds of Git or Mercurial repos.

Distributed vs. centralized VCS has very little directly to do with many vs. monolithic repos. After all, git was originally developed for a project with a large monolithic repo. Distributed VCS and many small repos got popular around the same time, but that's partly coincidental (microservice architectures getting popular, npm community preferring extremely small libraries) and partly because of GitHub making it very cheap in money/time to have many git repos.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#126

Post by flukus »

joshuamorton wrote: post #106
flukus wrote: post #80

Earlier quoted context omitted.

> and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo That's a huge understatement. They haven't just slapped a few scripts on top of git/svn, they've created their own proprietary scm to manage all of this. They've thrown more at this beast than most companies will throw at their actual product. I'm also not convinced they haven't reinvented…

Your last paragraph doesn't sound like anything at Google. Most engineers will never use branches at all, and even fewer will use branches that merge into trunk (instead of away from it).

There is a set of code changes locally and those changes are bundled off to the test server to run the full test suite? That's a branch.

Now let's say I break a project sharing this code and because I'm not an expert in all 2 billion LoC and 3000 projects google is running I need to enlist some help in fixing what I broke. Presumably there is a way for the developers on that downstream project to pull in my change set? That's a shared branch.

Now assuming I can get all of these planets aligned correctly I'm going to need to take this set of changes and put it into the master version aren't I? That's merging my branch into trunk.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#127

Post by ebikelaw »

flukus wrote: post #96
robert-brown wrote: post #71

Earlier quoted context omitted.

Yes, that's what you do. And you commit the change when all the tests pass.

And that's insane, people don't scale like that. It's harder enough keeping your head around one large project let alone every project a company has that you might have to jump into at any point.

You're arguing for the nonexistence of something that obviously exists. There are tens of thousands of engineers at Google working in this manner on one of the largest codebases ever assembled.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#128

Post by shados »

Eridrus wrote: post #105
whack wrote: post #87

Earlier quoted context omitted.

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, al…

> The next morning, all your services end up broken as a result. I mean, this is the argument for having good integration tests. At some point someone has to figure out if the new code will break a system; if you don't have good integration tests you're basically left eyeballing the changes, and sure eyeballing changes can work fine in small teams, but at some point you need good tests. I did some work on a ranking s…

> I mean, this is the argument for having good integration tests.

Maybe companies like Google have very, very strong code hygiene, but at most places I've worked, sooner or later there's a project that had a tight deadline and someone thought it smart to cut corners on the tests. Or the test s are there but they're bad. Or incomplete. Or worse, some system was just too hard to test and not updated frequently enough, so it requires manual testing.

Having "eventual consistency" for this is quite nice. Push a breaking change, update what you can, run tests, speak with owners, get deployed what you can. Keep tab of what you couldn't. Then do what you have to do to get the stuff tested (even if it means manual) and gradually become consistent as these things get pushed to prod...and hopefully next time its easier.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#129

Post by YokoZar »

shub wrote: post #89
whack wrote: post #87

Earlier quoted context omitted.

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, al…

> Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. Someone makes a commit to library code and production magically breaks? How does that happen?

> Someone makes a commit to library code and production magically breaks? How does that happen?

With insufficient tests and broken release processes.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#130

Post by Tloewald »

bedros wrote: post #112
adamrt wrote: post #69

Earlier quoted context omitted.

We moved to a monorepo about 2 years ago and it has been nothing but success for us. We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all…

how do you create branches in mono repo? for example I want to use branch rev5 from project A and rev3 from project B how I do that in a mono repo, I could not do it in HG, but sure about GIT

Can’t you create a branch and merge the two branches you are interested in into that?
Post reply on HN