Live data from Hacker News

Google stores billions of lines of code in a single repository (2016) [pdf]

dl.acm.org

131–140 of 209 posts

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#131
post #130
post #126

Earlier quoted context omitted.

Almost. We had a UI library on Android that was stuck on an alpha version of the library for three or so years after the library had shipped. Upgrading the library broke many tests across the org, and no one wanted to own going in and getting each team to fix it. Eventually, the library had a v2 release, and people started to care about being able to use it. Ultimately, they just forked the current release and append…

> The monorepo works for Google Does it? Or is it stopping Google from supporting products which only make millions in revenue because of the massive burden of continually updating?

Oh geez, that's an entirely different can of worms that isn't related to the monorepo.

Most products at Google are not dropped because the monorepo makes it difficult for them to support - and I'm not sure how it would or how you got to that association. Also, plenty of products that are killed are not in the monorepo.

They are usually dropped due to a mix of things, but a big part is just better product management.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#132
post #44

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

It’s not really even a true monorepo. Little known feature - there is a versions map which pins major components like base or cfs. This breaks monorepo abstraction and makes full repo changes difficult, but keeps devs of individual components sane.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#133
post #51
post #44

Earlier quoted context omitted.

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Google's software mostly uses dependencies already in the google monorepo, so these issues don't crop up. The person/team working on library changes have to ensure that nothing breaks, or the downstream users are notified early on. Don't think this would apply to many companies.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#134

Earlier quoted context omitted.

> The next deploy took our service down. How would multi-repo change this? A dependency updated, and code broke, and the new version was broken—but you update dependencies in multi-repo anyway, and deployments can be broken anyway. I don’t see how multi-repo mitigates this. > It encourages poor API contracts because it lets anyone import any code in any service arbitrarily. This has nothing at all to do with monorepo…

> How would multi-repo change this? A dependency updated, and code broke, and the new version was broken—but you update dependencies in multi-repo anyway, and deployments can be broken anyway. I don’t see how multi-repo mitigates this. In a multi-repo world, I control the repo for my own service. For a business-critical service in maintenance mode (with no active feature development), there's no reason for me to upgr…

> In a multi-repo world, I control the repo for my own service. For a business-critical service in maintenance mode (with no active feature development), there's no reason for me to upgrade the dependencies. Code changes are the #1 cause of incidents; why fix something that isn't broken?

This is now the “what is code rot?” discussion, which is an incredibly deep and nuanced discussion and I’m not going to do it justice here.

Just to pick an example—if you have an old enough version of your SSL library, it won’t be able to connect to modern web servers, depending on the configuration (no algorithms in common). If you have old database software, maybe it won’t work with the centralized backup service you have. If your software is stuck on 32-bit, maybe you run out of RAM, or maybe the vendors stop supporting the hardware you need. If you need old development tools to build your software, maybe the developers won’t be able to make changes in the future when they actually become necessary. What if your code only builds with Visual Studio 6.0, and you can’t find a copy, and you need to fix a defect now?

As much as I like the idea of building software once and then running the same version for an eternity, I prefer the idea of updating dependencies and spending some more time on maintenance. I advocate for a certain minimum amount of code churn. If the code churn falls too low, you end up with getting blind-sided by problems and don’t have any expertise to deal with it. My personal experience is that the amount of time you spend on maintenance with changing dependencies doesn’t have to be burdensome, but it’s project-dependent. Some libraries you depend on will cause headaches when you upgrade, some won’t. Good developers know how to vet dependencies.

If you really need an old version of a dependency, you can always vendor an old version of it.

If you can’t afford to put developers on maintenance work and make changes to old projects, then maybe those projects should move from maintenance to sunset.

> that there exists a team or group of volunteers to maintain such a build system across the entire repo

Bazel doesn’t require a whole team. My personal experience is that a lot of teams end up with one or two people who know the build system and know how to keep the CI running, and I don’t think this changes much with Bazel.

Bazel is actually very good at isolating build failures. You can do all sorts of things that break the build and it will only affect certain targets. It is better at this than a lot of other tools.

> underlying technical problem of low-signal PRs

I honestly don’t see a few low-signal PRs as a problem. PRs are not there to provide “signal”, they are just there to logically group changes.

The kind of PRs that I see go by in monorepos are things like “library X has deprecated interface Y, and the maintainers of X are migrating Y to Z so they can remove Y later”. Maybe your experience is different.

I do think that owners should not feel that they need to carefully review every PR that touches the code that they own. This is, IMO, its own anti-pattern—your developers should build processes that they trust, monitor the rate at which defects get deployed to production, and address systemic problems that lead to production outages. Carefully reviewing each PR only makes sense in certain contexts.

If you’re working in some environment where you do need that kind of scrutiny for every change you make, then you are probably also in an enviroment where you need to apply that scrutiny to dependencies. Maybe that means your code has fewer dependencies and relies more on the stdlib.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#135

Earlier quoted context omitted.

Bazel is not failing in the open source world though

Dont get me wrong, I love Bazel. But tooling for it to build multi-language multi-billion line monorepos doesn’t exist outside big companies.

[deleted]

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#136

There's a lot of love for monorepos nowadays, but after more than a decade of writing software, I still strongly believe it is an antipattern. 1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between ser…

Code Monoliths make just about as much sense as Runtime Monoliths, that is to say, if you are splitting your project into different micro-services, you can split your code base into different repositories too.

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#137
post #132
post #44

Earlier quoted context omitted.

Context: Staff Eng @ Google for 7+ years 1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump. 2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfac…

It’s not really even a true monorepo. Little known feature - there is a versions map which pins major components like base or cfs. This breaks monorepo abstraction and makes full repo changes difficult, but keeps devs of individual components sane.

This was done away with years ago. Components are no more.

There are still a couple of things that develop on long lived dev branches instead of directly at head, but my personal opinion is the need for those things to do that is mostly overstated (and having sent them cls in the past, it's deeply annoying).

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#139
post #84
post #51

Earlier quoted context omitted.

> any breaking changes should be dealt with in the same change as the version bump Does this mean that some things will never get updated, as the effort required is impossibly high?

Effort to update something is high because there's a lot of code, not because it's in a monorepo. Updating the same code scattered across multiple repositories takes as much work in the best case. More realistically, some copy of the same code will stay unupdated because the cost to track down every repository in the company is too much.

But with a multi-repo, its possible to e.g. upgrade the dependency just for a single service that has an immediate need for the upgrade, isn't it?

Re: Google stores billions of lines of code in a single repository (2016) [pdf]

#140

Earlier quoted context omitted.

Even so, the cost is often outrageously high. No to mention that if you're the first team to import a third_party library, you own it and other teams can add arbitrary cost to you updating it. You have to be very aggressive with visibility and SLAs to work around this.

You're basically just describing all the pain with pulling in a dependency regardless of monorepo or not. If the third party dependency does not add enough value to justify the cost then don't add it.

In a multi-repo setup you can upgrade gradually though, tackling the services that need the upgrade the most first. Can you do that in a monorepo setup?
Post reply on HN