Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

11–20 of 217 posts

Re: Facebook hit git performance issue on large repository

#11

That's projected growth for two of their projects. Sounds like they have something brewing... Still amazed that breaking it up would do more harm than good when the code isn't even written yet...

I read it as them being unable to break up a project, and the repo being a projection of future commits to a project that can't be split up.

Re: Facebook hit git performance issue on large repository

#12
post #10

While I'd be interested in seeing this issue further unfold, just the prospect of a 1.3M-file repo gives me the creeps. I'm not sure what the exact situation at Facebook is with this repository, but I'm positive that if they had to start with a clean slate, this repo would easily find itself broken up into at least a dozen different repos. Not to mention the fact that if _git_ has issues dealing with 1.3M files, I wo…

Pretty sure Perforce performs fine with that.

Re: Facebook hit git performance issue on large repository

#13
post #10

While I'd be interested in seeing this issue further unfold, just the prospect of a 1.3M-file repo gives me the creeps. I'm not sure what the exact situation at Facebook is with this repository, but I'm positive that if they had to start with a clean slate, this repo would easily find itself broken up into at least a dozen different repos. Not to mention the fact that if _git_ has issues dealing with 1.3M files, I wo…

If you read further down the thread they say that that's already had the non-interlinked files split out. What they've got left isn't easily broken up.

Re: Facebook hit git performance issue on large repository

#14

This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…

Given that Facebook is compiled into a single 1 GB executable, a git repo with 1.3 M files doesn't really surprise me.

Re: Facebook hit git performance issue on large repository

#16
post #10

While I'd be interested in seeing this issue further unfold, just the prospect of a 1.3M-file repo gives me the creeps. I'm not sure what the exact situation at Facebook is with this repository, but I'm positive that if they had to start with a clean slate, this repo would easily find itself broken up into at least a dozen different repos. Not to mention the fact that if _git_ has issues dealing with 1.3M files, I wo…

If you read further down the thread they say that that's already had the non-interlinked files split out. What they've got left isn't easily broken up.

Best argument I've ever seen for not wanting to work at Facebook... wow that's a lot intertwined spagetti code.

Our source repo at work (a C++ compiler with full commit history going back to the early 90s...) is smaller and more componentized!

Re: Facebook hit git performance issue on large repository

#17
post #12
post #10

While I'd be interested in seeing this issue further unfold, just the prospect of a 1.3M-file repo gives me the creeps. I'm not sure what the exact situation at Facebook is with this repository, but I'm positive that if they had to start with a clean slate, this repo would easily find itself broken up into at least a dozen different repos. Not to mention the fact that if _git_ has issues dealing with 1.3M files, I wo…

Pretty sure Perforce performs fine with that.

Well, of course that at some specific scale, you're gonna start to have trouble with any DVCS maintaining a complete local copy of such a huge repository.

It's even worse that just disk space and performance issues.

I can totally imagine a huge, busy repository where by the time you've pulled and rebased/merged your stuff, the repo has already been committed to again, invalidating your fast-forward commit and forcing you to pull again and again before you have any chance of pushing back your changes.

This is an inherent problem with DVCS that just can't be solved (trivially) when working on huge repositories that span millions of files and involve thousands of developers.

Re: Facebook hit git performance issue on large repository

#18
post #10

While I'd be interested in seeing this issue further unfold, just the prospect of a 1.3M-file repo gives me the creeps. I'm not sure what the exact situation at Facebook is with this repository, but I'm positive that if they had to start with a clean slate, this repo would easily find itself broken up into at least a dozen different repos. Not to mention the fact that if _git_ has issues dealing with 1.3M files, I wo…

A lot of big companies have repos 10 or 100 times that size. With tens of millions of files, sometimes up to 100 gigs or more of data under source control.

Re: Facebook hit git performance issue on large repository

#19
http://thread.gmane.org/gmane.comp.version-control.git/18977...

They keep every project in a single repo, mystery solved.

Edit:

> We already have some of the easily separable projects in separate repositories, like HPHP.

Yeah, because it makes no sense, it's C++. They probably use for everything PHP i assume then. Is there no good build management tool for it?

Re: Facebook hit git performance issue on large repository

#20
Others have tried and keep throwing more and more smart people at the problem they just shouldn't have.

MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce.

Google with all their source code in one Perforce repo.

Facebook will be on perforce before we know it.

The solution is an internal Github, not one giant project.

Post reply on HN