Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

201–210 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#201

Post by shiift »

tzhenghao wrote: post #143

Having worked at different companies adopting both monorepo and the multiple repos approach, I find monorepo a better normalizer at scale in consolidating all "software" that runs the company. Just like what many commenters here have mentioned, the monorepo approach is a forcing function on keeping compatibility issues at bay. What you don't want is to end up in a situation where teams reinvent their own wheels inste…

I really liked that talk! Lots of relevant information and I can definitely relate, working at a Amazon. Wouldn't say that we are hurt by all of the same problems (we have solutions that work very well for some of them), but we definitely are aware of them.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#202

Post by Khoth »

tiglionabbit wrote: post #108
favorited wrote: post #95

Earlier quoted context omitted.

Someone from FB did a really cringe-inducing presentation a few years ago about how "X can't handle our scale" (I think the predicate was iOS, but they went into IDEs and SCM systems). They had to pull the video and slides because it was so bad.

XCode also had issues working with large repos. Perhaps they were talking about that?

IIRC their iPhone app had about 20000 classes for some insane reason, and the system didn't handle that well.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#203

Post by stevesimmons »

My company has a 50m LOC Python codebase in a monorepo. It works really well, given the rate of change of thousands of developers globally. That is only possible because of the significant investment in devtools, testing and the deployment infrastructure.

Here is "Python at Massive Scale", my talk about it at PyData London earlier this year:

https://youtu.be/ZYD9yyMh9Hk

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#204

Post by perlgeek »

anyfoo wrote: post #59
whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" the…

I thought that Google deploys new versions gradually (first to 1% of users, and if that doesn't show errors, to more and more). Which implies that there are at least two version of an application or service running.

How does that work when you don't keep APIs stable, at least at the service boundaries?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#205

Post by flukus »

senozhatsky wrote: post #198
glandium wrote: post #160

Earlier quoted context omitted.

The reason why y2038 problem looked so simple to OpenBSD has little to do with "monolithic repository" and everything to do with "happy to break kernel ABI compatibility". You're saying as much yourself. Monolithic repository might have been a tool that helped enforce it, but that's not what made it happen. It's the decision that ABI could be broken that did. And that's also why it hasn't happened in Linux yet. Even…

> Monolithic repository might have been a tool that helped enforce it, > but that's not what made it happen. It's the decision that ABI could > be broken that did. Well, there are probably some subtle details which I'm missing, and may be you are totally right. The way it looks to me is as follows: They are "happy to break kernel ABI compatibility" because the repository is monolithic - they break ABI, they immediate…

> There is no "transitional" stage, when the kernel is already patched, but no user space apps are ready for those changes yet. It all happens at once.

What about third party apps? It's not a fully self contained system, there are binaries out there running on openBSD that the openBSD devs have never heard of, and they were broken by the change.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#206

Post by gefh »

mlthoughts2018 wrote: post #93

One of my former managers had worked a long time at Google and was present for the advent of Google’s in-house tooling developed around their monorepo. His account was that it was basically accidental, at first resulting from short term fire drills, and then creating a snowball effect where the momentum of keeping things in the Perforce monorepo and building tooling around it just happened to be the local optimum, an…

Do you wonder whether he is a reliable narrator?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#207

Post by georgewfraser »

timkrueger wrote: post #189
georgewfraser wrote: post #146

Earlier quoted context omitted.

I kinda disagree, we’re a dev team of 30, 3.5 years in, 150k lines of code and we’ve always had a monorepo. We had to switch from maven to bazel after about 2 years because test times got out of control; bazel has been about 50% more annoying than maven but the incremental builds work perfectly.

Interesting. Do you have wrote something about that migration?

No, though that would be a good blog post. We tried to make multi-module maven work for a while, eventually gave up and wrote some scripts that would convert maven to bazel, using many assumptions that applied only to our particular case. We did the cutover in one day but kept maven around for a couple weeks in case we decided to bail on bazel. It worked out; we even found CircleCI works great. I would say the weak link in the bazel ecosystem is the IntelliJ plugin, which is very functional but also very slow.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#208

Post by senozhatsky »

flukus wrote: post #205
senozhatsky wrote: post #198

Earlier quoted context omitted.

> Monolithic repository might have been a tool that helped enforce it, > but that's not what made it happen. It's the decision that ABI could > be broken that did. Well, there are probably some subtle details which I'm missing, and may be you are totally right. The way it looks to me is as follows: They are "happy to break kernel ABI compatibility" because the repository is monolithic - they break ABI, they immediate…

> There is no "transitional" stage, when the kernel is already patched, but no user space apps are ready for those changes yet. It all happens at once. What about third party apps? It's not a fully self contained system, there are binaries out there running on openBSD that the openBSD devs have never heard of, and they were broken by the change.

Good point. The way BSD people look at this issue - 3rd party apps are sort of out of consideration.

I think they actually consider it to be self-contained.

[UPD] But I haven't checked how NetBSD handled y2038. It might be that they didn't break the ABI. In some parts, I think, OpenBSD/FreeBSD/NetBSD just converted time_t to an unsigned 32-bit int.

-ss

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#209

Post by vkou »

u801e wrote: post #119
vkou wrote: post #70

Earlier quoted context omitted.

You can soft-deprecate the old code path, and communicate that warnings will turn into errors at some future date. You can send your pull request to the affected team leads, and request that they approve it, once they make changes on their end. I mean, the alternative is that you have 17 different projects, each using one of five different versions of the common code. Heaven forbid one of them makes an incorrect assu…

If the common code is a versioned package, then each of the 17 different projects could update their code to handle breaking changes in the common package independently and update the version dependency after thorough testing.

You can have versioned packages inside a mono-repository, too, though. /common_libs/foo_lib_v1.13/, /common_libs/foo_lib_v1.14/, etc.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#210

Post by flukus »

brazzledazzle wrote: post #72
flukus wrote: post #58

Earlier quoted context omitted.

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

I don't have a horse in this race but if you have, say, a security issue and that needs to propagate downstream where does your responsibility end in that situation? Do you try to track down the dependencies and open issues in their trackers? Or maybe a more common problem is a change to a library that consumes an API that's changing so updating the library has a drop dead date.

> I don't have a horse in this race but if you have, say, a security issue and that needs to propagate downstream where does your responsibility end in that situation?

This is an issue that needs to be managed, from the systems I've seen it tends to be managed poorly, that's in both monoish repos and multi-repo setups as well as everyone using third party packages. I don't think committing everything to trunk is a good way to resolve it though, they only upside to this approach is that it might force you to resolve it.

What I have to deal with much more frequently is the opposite problem, we have an urgent update that will break several things but has to be deployed for one dependent binary ASAP and fixing the rest of the universe first is not an option.

Worst case it might create some security issues, something that should be a breaking change getting kludged into a new breaking change but still being broken.

Post reply on HN