Facebook hit git performance issue on large repository
thread.gmane.org
Facebook hit git performance issue on large repository
1–10 of 217 posts
Re: Facebook hit git performance issue on large repository
#2[deleted]
Re: Facebook hit git performance issue on large repository
#3This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components.
I'd be very interested to see some benchmarks on their current VCS solution for repositories of this scale.
Re: Facebook hit git performance issue on large repository
#4This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…
From a followup post: “We already have some of the easily
separable projects in separate repositories, like HPHP. If we could split our largest repos into multiple ones, that would help the scaling issue. However, the code in those repos is rather interdependent and we believe it’d hurt more than help to split it up, at least for the medium-term future. We derive a fair amount of benefit from the code sharing and keeping things together in a single repo, so it's not clear when it’d make sense to get more aggressive splitting things up.”
Re: Facebook hit git performance issue on large repository
#5I hope these guys do take the route of developing a large-scale performant patch.
Git as so many interesting uses at scale as just a tool that navigates and tracks DAGs over time.
Re: Facebook hit git performance issue on large repository
#6This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…
Remember that this is a synthetic repository.
Re: Facebook hit git performance issue on large repository
#7That's projected growth for two of their projects. Sounds like they have something brewing...
Still amazed that breaking it up would do more harm than good when the code isn't even written yet...
Re: Facebook hit git performance issue on large repository
#8This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…
He notes that these repositories are somewhat broken up already, and wants to keep them together.
There are good reasons to keep code in one repository; particularly, git's submodule support has a number of nasty interface tradeoffs; I wouldn't say it breaks git, but you have to keep a clear understanding of all your submodules in your head when you have a lot of them.
OK, it pretty much breaks git to have submodules that are interdependent. I know this because I am currently moving one of my organizations off this exact plan -- it's the opposite of useful and speedy to have to worry about versions across a large number of backend / frontend repositories.
It is MUCH easier and therefore better for developers to put them together, and release together.
Re: Facebook hit git performance issue on large repository
#9This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…
Not sure if it's still the case, but Google hosts all their internal source code on a modified version of perforce, so they essentially have everything in one repo.
Re: Facebook hit git performance issue on large repository
#10While I'd be interested in seeing this issue further unfold, just the prospect of a 1.3M-file repo gives me the creeps.
I'm not sure what the exact situation at Facebook is with this repository, but I'm positive that if they had to start with a clean slate, this repo would easily find itself broken up into at least a dozen different repos.
Not to mention the fact that if _git_ has issues dealing with 1.3M files, I wonder what other (D)VCS they're thinking of as an alternative that would be more performant.