Facebook hit git performance issue on large repository • Hacker News Board

Re: Facebook hit git performance issue on large repository

#111

Post by chubot » Sat, Feb 04, 2012, 12:40 AM UTC

forrestthewoods wrote: post #90

kenrik wrote: post #69

Earlier quoted context omitted.

With that in mind it seems like there is a market for a git replacement for these huge repos.

It's called Perforce and anyone dealing with binary files has been using it for years.

And also paying Perforce fistfuls of cash in licensing fees. I hear that Perforce is a quite a small company, and the founder wrote the lion's share of the code a couple decades ago.

I think they are probably on par with craigslist in profits per employee (i.e. much higher than Google or Facebook. Interestingly I think Facebook has about 1/10 the employees of Google with 1/10 the profits -- off the top of my head feel free to correct -- so I don't think they blew it out of the park with their IPO filing).

Re: Facebook hit git performance issue on large repository

#112

Post by mindcrime » Sat, Feb 04, 2012, 12:47 AM UTC

EricBurnett wrote: post #101

mindcrime wrote: post #94

Earlier quoted context omitted.

Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstatio…

is anybody going to parse through all of that? Yes, in fact. We have some great tools that give us full search over our entire codebase (think Google Code Search), and you can add a dependency on a piece of code without needing to have it on your workstation already. The magic filesystem our build tools use knows where to get it and can do so on demand. Combined with good code location conventions, an overall attitud…

Awesome, glad to hear you guys take reuse so seriously. I'm a little surprised, only because - in my experience - so few organizations put in the effort that you guys do.

Re: Facebook hit git performance issue on large repository

#113

Post by pnathan » Sat, Feb 04, 2012, 12:51 AM UTC

jrockway wrote: post #63

Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful. So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase,…

> Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code,

Can you expand on this? I would love to talk more about the "well known" part, I've never run across it before. I am a maintainer (tools guy actually) of a hg repo with about 120 subrepos, and the whole approach with subrepos is something that we're not thrilled about. Oh, and if you want to communicate via email, I'd be up for that too.

Re: Facebook hit git performance issue on large repository

#114

Post by aidenn0 » Sat, Feb 04, 2012, 12:53 AM UTC

losvedir wrote: post #28

Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases. So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebo…

The linux kernel is several orders of magnitude smaller. They are talking about 1.3 million files totalling nearly 10GB for the working tree. My kernel checkout has 39 thousand files totaling 489MB.

Re: Facebook hit git performance issue on large repository

#115

Post by [deleted] » Sat, Feb 04, 2012, 12:55 AM UTC

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

[deleted]

Re: Facebook hit git performance issue on large repository

#116

Post by jmccaffrey » Sat, Feb 04, 2012, 12:59 AM UTC

ramanujan wrote: post #109

This looks like it could be of assistance: http://source.android.com/source/version-control.html Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow. Repo is not meant to replace Git, only to make it easier to work with Git in the context of Andr…

Having worked with repo professionally, I'm not a fan. You lose simple ability to track dependencies across repositories or even revert to a previous consistent point in time without diligent tagging. Even with good tags, restructuring your project setup and changing your repo manifest can still break your ability to go back in time.

Re: Facebook hit git performance issue on large repository

#117

Post by nbm » Sat, Feb 04, 2012, 1:10 AM UTC

moe wrote: post #104

nbm wrote: post #78

Earlier quoted context omitted.

The example is synthetic, so don't worry too much about the implications. It is useful to keep in mind that Facebook isn't just the front-end (and isn't just code, also images, configuration, and so forth). Just talking about open source stuff, Facebook also generates code like Cassandra, Hive (data warehousing application), Phabricator (a code review and lifecycle tool), HipHop for PHP (the translator/compiler, the…

And all of that must live in a single repository... because?

It doesn't live in a single repository. The commenter I was replying to mentioned "Facebook and all its peripheral development" and a number of lines of code. I wanted to give him a little insight into what sort of things all the peripheral development might include, since it isn't obvious.

Re: Facebook hit git performance issue on large repository

#118

Post by SomeCallMeTim » Sat, Feb 04, 2012, 1:12 AM UTC

nbm wrote: post #72

SomeCallMeTim wrote: post #67

Earlier quoted context omitted.

None of what you mention here precludes breaking up the code into many smaller repositories, and then having them all linked together in one super-repository. Then tags at the super-repository level can record the exact state of all submodules. It's not about not checking the other modules out; you can make this the standard behavior, sure. Instead it's about having git manage reasonable sized blocks of the code base…

I'm not sure this would work - if an operation needs you to stat() a file for every file in the repo (for example), whether it is 10k files in 1 repo or 1k files each in 10 repos will probably just as bad?

An operation in git makes you stat() each file in the current repo -- so things like check-ins and local operations would be done 100% in the current repo.

Any time you were pulling the entire repo tree, it could be slow, yes. But assuming people are only working in one or a small number of repos at once, you can imagine a workflow that didn't involve nearly so many operations on the entire tree.

Re: Facebook hit git performance issue on large repository

#119

Post by EricBurnett » Sat, Feb 04, 2012, 1:12 AM UTC

EricBurnett wrote: post #101

mindcrime wrote: post #94

Earlier quoted context omitted.

Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstatio…

is anybody going to parse through all of that? Yes, in fact. We have some great tools that give us full search over our entire codebase (think Google Code Search), and you can add a dependency on a piece of code without needing to have it on your workstation already. The magic filesystem our build tools use knows where to get it and can do so on demand. Combined with good code location conventions, an overall attitud…

Reply to mindcrime sibling post

I've seen that before myself at other companies, and it's a shame. A healthy codebase is an investment in the future - if you're not taking the time to cultivate it you're sacrificing long term usability for short term gains. The larger the codebase the more difficult the task, of course, but for us that's just an excuse to solve the next hard problem :).

One more good link on the topic: our use of Clang to find and fix bugs in our existing codebase, as we find new classes of 'gotchas'. http://google-engtools.blogspot.com/2011/05/c-at-google-here...

Re: Facebook hit git performance issue on large repository

#120

Post by thurn » Sat, Feb 04, 2012, 1:12 AM UTC

jrockway wrote: post #63

Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful. So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase,…

Facebook uses Subversion for its trunk, actually, and just gets developers to use git-svn. This issue is primarily a problem because git-svn is a lot more serious about replicating the true git experience (keep everything local) than Google's p4-git wrapper is. They really just need to be a little less religious about keeping everything local.