Facebook hit git performance issue on large repository • Hacker News Board

Re: Facebook hit git performance issue on large repository

#121

Post by rachelbythebay » Sat, Feb 04, 2012, 1:13 AM UTC

mikeocool wrote: post #35

sek wrote: post #29

Earlier quoted context omitted.

Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.

No, literally the entire codebase for all of their products is in one Perforce repo. Ashish Kumar, manager of the Engineering Tools team, mentions it in this presentation: http://www.infoq.com/presentations/Development-at-Google

The kernel? Android? Some other spooky stuff involving the pest control guy who's holding a big rubber mallet when you fail a unit test?

Are you sure about that?

Re: Facebook hit git performance issue on large repository

#122

Post by rogerbinns » Sat, Feb 04, 2012, 1:13 AM UTC

jpdoctor wrote: post #70

$100B company, maybe they can afford to put some people onto solving this for the open software community (and put the solution into the open), especially since nobody else in the community seems to have this problem.

If you proposed a good solution I'm sure they'd be happy to provide time and money and open source the result. But most of the responses aren't even that there is a solution - they say to split the repository into smaller pieces and spend time and money internally having their internal developers deal with that.

A good solution will benefit everyone who uses git. Codebases get larger over time. There is more forking and experimentation. More spoken languages can be supported. More computer languages can be interfaced. The O(n) operations becoming less than that will benefit you in the future as your code grows.

Re: Facebook hit git performance issue on large repository

#123

Post by Terretta » Sat, Feb 04, 2012, 1:14 AM UTC

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

> waiting on disk I/O

Out of curiosity, why are these benchmarks using regular disk and flash disk? At only 15 GB, what happens using ram disk? Sure SSD is fast, but for these things it's still really slow.

Re: Facebook hit git performance issue on large repository

#124

Post by nostrademons » Sat, Feb 04, 2012, 1:26 AM UTC

ghshephard wrote: post #22

clord wrote: post #16

Earlier quoted context omitted.

Best argument I've ever seen for not wanting to work at Facebook... wow that's a lot intertwined spagetti code. Our source repo at work (a C++ compiler with full commit history going back to the early 90s...) is smaller and more componentized!

That C++ compiler is a single product (okay, you might have built a linker, and an assembler as well - say 3-5 products). In even medium enterprises (say, 500 employees, about 250 developers) you might have upwards of 35 different products, each of which with a 5-6 year active history. Enterprise source control can be ugly - particularly if you have non-text resources (Art, Firmware Binaries, tools) that need to be c…

There're costs and benefits both ways. AFAIK, Microsoft and Amazon both use the separate repositories model, and Google and Facebook use a single large repository. Most people I know that have worked at both of these styles prefer the Google/Facebook style.

The biggest advantage of a single repository is pretty intangible - it's cultural. When anyone can change anything or can use any code, people feel like the whole company belongs to them, and they're responsible for Google/Facebook's success as a whole. People will spontaneously come together to accomplish some user need, and they can refactor to simplify things across component boundaries, and you don't get the sort of political infighting that tends to plague large organizations where people in subprojects never interact with each other.

I think if it were my company, I'd want the single repository model, but there need to be tools and practices to manage API complexity. I dunno what those tools & practices would look like; there are some very smart people in Google that are grappling with this problem though.

Re: Facebook hit git performance issue on large repository

#125

Post by nostrademons » Sat, Feb 04, 2012, 1:32 AM UTC

rachelbythebay wrote: post #121

mikeocool wrote: post #35

Earlier quoted context omitted.

No, literally the entire codebase for all of their products is in one Perforce repo. Ashish Kumar, manager of the Engineering Tools team, mentions it in this presentation: http://www.infoq.com/presentations/Development-at-Google

The kernel? Android? Some other spooky stuff involving the pest control guy who's holding a big rubber mallet when you fail a unit test? Are you sure about that?

Kernel/Android/Chrome/basically anything open-source is different. If the code is going to be open-sourced, it can't have dependencies on proprietary code anyway.

Re: Facebook hit git performance issue on large repository

#126

Post by DannoHung » Sat, Feb 04, 2012, 1:38 AM UTC

Multiple people in this conversation section have asserted that code sharing is way easier when all the code is in a single repo, but from my understanding of sub-modules, it would be a fairly simple matter of setting up your pre/post-commit hooks to update submodules to a branch automatically and get useful company wide change atomicity (after all, changes should only propagate between teams/projects once they have some stability).

Putting aside the question of whether or not an enormous singular repo can be broken up intelligently into modular projects, is there something about the submodule approach that makes it a uniquely unsuitable way for sharing changes amongst projects?

Re: Facebook hit git performance issue on large repository

#127

Post by nbm » Sat, Feb 04, 2012, 1:39 AM UTC

thurn wrote: post #120

jrockway wrote: post #63

Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful. So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase,…

Facebook uses Subversion for its trunk, actually, and just gets developers to use git-svn. This issue is primarily a problem because git-svn is a lot more serious about replicating the true git experience (keep everything local) than Google's p4-git wrapper is. They really just need to be a little less religious about keeping everything local.

Just a minor clarification - some of the repositories use Subversion (as mentioned in the "Push" tech talk), and some are pure git.

Re: Facebook hit git performance issue on large repository

#128

Post by sateesh » Sat, Feb 04, 2012, 1:42 AM UTC

forrestthewoods wrote: post #90

kenrik wrote: post #69

Earlier quoted context omitted.

With that in mind it seems like there is a market for a git replacement for these huge repos.

It's called Perforce and anyone dealing with binary files has been using it for years.

And also there is 'Rational Clearcase', another centralized version control system.

Re: Facebook hit git performance issue on large repository

#129

Post by huytoan_pc » Sat, Feb 04, 2012, 1:49 AM UTC

jlarocco wrote: post #27

wmf wrote: post #14

Earlier quoted context omitted.

Given that Facebook is compiled into a single 1 GB executable, a git repo with 1.3 M files doesn't really surprise me.

What? Do you have a reference for that?

Here you go: http://www.facebook.com/note.php?note_id=10150121348198920

"We can build a binary that is more than 1GB (after stripping debug information) in about 15 min, with the help of distcc. Although faster compilation does not directly contribute to run-time efficiency, it helps make the deployment process better."

Re: Facebook hit git performance issue on large repository

#130

Post by twinge » Sat, Feb 04, 2012, 1:49 AM UTC

SomeCallMeTim wrote: post #67

nbm wrote: post #59

Earlier quoted context omitted.

I wouldn't say that a large repository implies that the code is highly-coupled. There are advantages for keeping certain code together in a single repo. Being able to easily discover users of functions of a library, being able to "transactionally" commit an update to a library (or set of libraries) and the code that uses it, being able to do code review over changes of code in various places, being able to discover i…

None of what you mention here precludes breaking up the code into many smaller repositories, and then having them all linked together in one super-repository. Then tags at the super-repository level can record the exact state of all submodules. It's not about not checking the other modules out; you can make this the standard behavior, sure. Instead it's about having git manage reasonable sized blocks of the code base…

Three big problems with a split up codebase:

1) Instead of doing one large release every week (which facebook does: http://www.facebook.com/video/video.php?v=10100259101684977) you now have dozens or hundreds of smaller releases, a lot more heterogeneity to test for.

2) If you have inter-dependencies on modules you have to grapple with the "diamond dependency" problem. Say module A depends on module B and C, and suppose that module B also depends on C. However, module B depends on C v2.0 but A depends on C v1.0. If they're all split across repositories it's not possible to update a core API in an atomic commit.

3) Now you rely on changes being merged "up" to the root and then you have to merge it "down" to your project. This is one of the reasons Vista was such a slow motion train wreck: http://moishelettvin.blogspot.com/2006/11/windows-shutdown-c... -- kernel changes had to be merged up to the root, then down to the UI, requiring months of very slow iterations to get it right.