Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

Post reply on HN
Print view View original

81–90 of 293 posts

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#81

Post by senozhatsky »

Well, it's not so uncommon. For instance, OpenBSD, NetBSD repos are sort of monolithic. And, believe it or not, there are some advantages. For instance, let's take a look at OpenBSD 5.5 [0] release notes:

> OpenBSD is year 2038 ready and will run well

> beyond Tue Jan 19 03:14:07 2038 UTC

OpenBSD 5.5 was released on May 1, 2014. While Linux is still "not quite there yet" y2038-wise. y2038 is a very complex issue, while it may look simple - time_t and clock_t should be 64-bit. This requires changes both on the kernel -- new sys-calls interfaces [stat()], new structures layouts [struct stat], new sizeof()-s, etc. -- and the user space sides. This, basically, means ABI breakage: newer kernels will not be able to run older user space binaries. So how did OpenBSD handle that? The reason why y2038 problem looked so simple to OpenBSD was a "monolithic repository". It's a self-contained system, with the kernel and user space built together out of a single repository. OpenBSD folks changed both user space and kernel space in "one shot".

IOW, a monolithic repository makes some things easier:

a) make a dramatic change to A

b) rebuild the world

c) see what's broken, patch it

d) while there are regressions or build breakages, goto (b)

e) commit everything

[0] http://www.openbsd.org/55.html?hn

[UPDATE: fixed spelling errors... umm, some of them]

-ss

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#82

Post by waterhouse »

hobs wrote: post #57
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

heh, thousands, its probably at least an OOM greater, if not two.

My brain sees OOM and thinks "out of memory", which might be applicable too.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#83

Post by tiglionabbit »

acomjean wrote: post #41
hobls wrote: post #28

I feel terrible for anyone who sees this and thinks, “ah! I should move to a monorepo!” I’ve seen it several times, and the thing they all seem to overlook is that Google has THOUSANDS of hours of effort put into the tooling for their monorepo. Slapping lots of projects into a single git repo without investing in tooling will not be a pleasant experience.

But what are the alternatives to the monorepo in git? All the ways of splitting code up and deploying multiple git repos for one project seem terrible.

Fun fact. I asked Facebook why they built their monorepo on Mercurial instead of Git. They said there were scaling issues in Git that made it unusable for large repos and the Git maintainers would not work with them to fix these issues. However, they were able to work with Mercurial to make it capable of holding their entire company in one repo.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#84

Post by tiglionabbit »

flukus wrote: post #58
ebikelaw wrote: post #52

Earlier quoted context omitted.

See, this is where your argument broke down for me. Once you’ve decided there is some library of common code, and assuming you factor out that code into another repo, you’ve just lost your ability to easily make breaking changes to the common code, which is something trivially easy to do in a monorepo. Why would you want that? It seems to me that if you have multiple projects sharing a base of common code then a mono…

> you’ve just lost your ability to easily make breaking changes to the common code I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible. With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do…

What you're proposing goes a step beyond multiple repos and into package versioning.

That is one viable workflow: Make a change to the common code and publish it as a new package version while allowing all existing code to continue to use the old package. Then, migrate other projects to the newer version of the dependency one by one.

Allowing multiple versions of the same code to exist in production at once adds complexity. It's a trade-off.

Also, if you're doing this with code that is ultimately webpacked to run in a web browser and you don't pay attention to the full tree of dependencies you're working with, there's a chance you end up loading two versions of the same library into a single web page, increasing the page weight and possibly causing incompatibilities in event handling.

Google prefers to simply have one master version of the entire company at a time.

I've spent a lot of time wondering which solution is the best and I'm still not sure.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#85

Post by mlinksva »

prepend wrote: post #64

I love these articles. Is there a wiki or collection of detailed descriptions of large company tech practices that isn’t marketing blargh. I read years ago about Google data ingest, locator process but neglected to bookmark so now can’t find the reference.

Me too. I don't know of a collection, but others can be found at https://ai.google/research/pubs/ https://research.fb.com/publications/ https://www.microsoft.com/en-us/research/search/?q&content-t... and similar (though only a small fraction give hints about at scale practices, and those would be neat to collect in one place).

Closely related to this post: just noticed a 2018 case study on Advantages and Disadvantages of a Monolithic Repository https://ai.google/research/pubs/pub47040

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#86

Post by haglin »

Google's handling of their source code makes me wanna work there.

I don't like distributed version control systems with hundreds of repositories spread out. It makes management more complicated. I understand this is a minority view, but that is my experience. It was easier to work in a single Perforce repository than hundreds of Git or Mercurial repos.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#87

Post by whack »

anyfoo wrote: post #59
whack wrote: post #51

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version.…

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" the…

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous.

Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result.

In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it.

In a monorepo world, you don't have a choice. You've been forcefully migrated as soon as the tooling team decides to make the change on their end. They had the best of intentions, but that doesn't always translate to a good outcome.

FWIW, I'm currently working at a large famous company that uses a monorepo. Color me not-impressed. I do think that having a single repository for an entire team/project is a good idea. Hundreds of different projects and teams who've never seen one another? Not so much.

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#88

Post by rhencke »

dmoy wrote: post #68
threeseed wrote: post #45

Earlier quoted context omitted.

Linux kernel is one functional piece of work though. Imagine if we combined KDE, Gnome, Linux Kernel, ZFS etc all in the one monorepo.

And gnucash, libreoffice, a couple copies of android, three other things that forked the linux kernel, and then all of apache to boot.

And we'll call it something crazy, like a Linux distribution!

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#89

Post by shub »

whack wrote: post #87
anyfoo wrote: post #59

Earlier quoted context omitted.

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" the…

It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, al…

> Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result.

Someone makes a commit to library code and production magically breaks? How does that happen?

Re: Why Google Stores Billions of Lines of Code in a Single Repository (2016)

#90

Post by nicodjimenez »

I have slight experience with both monorepos and smaller repos and I think they can both work. The advantage of smaller repos is that it forces different components to expose well designed API's. Bigger repos make sense for products and embedded software, smaller repos make sense for platforms build up of small services communicating on the internet.
Post reply on HN