I don't want to imagine the actual kind of code that requires 1.3M files to run.
[deleted]
Facebook hit git performance issue on large repository
21–30 of 217 posts
Re: Facebook hit git performance issue on large repository
#22Earlier quoted context omitted.
If you read further down the thread they say that that's already had the non-interlinked files split out. What they've got left isn't easily broken up.
Best argument I've ever seen for not wanting to work at Facebook... wow that's a lot intertwined spagetti code. Our source repo at work (a C++ compiler with full commit history going back to the early 90s...) is smaller and more componentized!
That C++ compiler is a single product (okay, you might have built a linker, and an assembler as well - say 3-5 products). In even medium enterprises (say, 500 employees, about 250 developers) you might have upwards of 35 different products, each of which with a 5-6 year active history.
Enterprise source control can be ugly - particularly if you have non-text resources (Art, Firmware Binaries, tools) that need to be checked in and version managed as well.
With all that said - I don't really understand why all the code is in a single repository. Surely a company of Facebook's size would experience some fairly great benefits from compartmentalization and published service interfaces. I guess I agree with the parent - sounds like a lot of intertwined spaghetti code. :-)
Re: Facebook hit git performance issue on large repository
#23This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…
What do Facebook and the National Institutes of Health have in common? I'm pretty sure this will end with Facebook building their own versioning system from scratch and give it some kitchsy name like "Retro".
Re: Facebook hit git performance issue on large repository
#24[deleted]
Re: Facebook hit git performance issue on large repository
#25http://thread.gmane.org/gmane.comp.version-control.git/18977... They keep every project in a single repo, mystery solved. Edit: > We already have some of the easily separable projects in separate repositories, like HPHP. Yeah, because it makes no sense, it's C++. They probably use for everything PHP i assume then. Is there no good build management tool for it?
> They keep every project in a single repo, mystery solved.
That's not true:
> It is based on a growth model of two of our current repositories (I.e., it's not a perforce import). We already have some of the easily separable projects in separate repositories, like HPHP. If we could split our largest repos into multiple ones, that would help the scaling issue. However, the code in those repos is rather interdependent and we believe it'd hurt more than help to split it up, at least for the medium-term future.
They already have multiple repositories, the stats they're doing there is based on "two of [their] current repositories" implying more than two.
Re: Facebook hit git performance issue on large repository
#26I don't think Git was designed to perform well with such a large repo. In this case, the best-practice is probably compartmentalizing the code and using Git submodules. The Git submodule interface is a little un-friendly, but I think it does work well for such large repos. I've been using submodules successfully for our development that tracks source files as well as binary assets.
Re: Facebook hit git performance issue on large repository
#27This was actually pretty fascinating to me. On one hand, I am astonished at how long it takes to perform seemingly trivial git operations on repositories at this scale. On the other hand, I'm utterly mystified that a company like Facebook has such monolithic repositories. Even back when I was using SVN a lot, I relied on externals and such to break up large projects into their smaller service-level components. I'd be…
Given that Facebook is compiled into a single 1 GB executable, a git repo with 1.3 M files doesn't really surprise me.
What? Do you have a reference for that?
Re: Facebook hit git performance issue on large repository
#28Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases.
So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebook is more massive even than the linux kernel, or there's some inherent design in git that works better for kernel work than web work?
Re: Facebook hit git performance issue on large repository
#29Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce. Google with all their source code in one Perforce repo. Facebook will be on perforce before we know it. The solution is an internal Github, not one giant proje…
Google has everything in one Perforce repo? You mean the search engine, do you?
I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.
Re: Facebook hit git performance issue on large repository
#30http://thread.gmane.org/gmane.comp.version-control.git/18977... They keep every project in a single repo, mystery solved. Edit: > We already have some of the easily separable projects in separate repositories, like HPHP. Yeah, because it makes no sense, it's C++. They probably use for everything PHP i assume then. Is there no good build management tool for it?
> They keep every project in a single repo, mystery solved. That's not true: > It is based on a growth model of two of our current repositories (I.e., it's not a perforce import). We already have some of the easily separable projects in separate repositories, like HPHP. If we could split our largest repos into multiple ones, that would help the scaling issue. However, the code in those repos is rather interdependent…
Why would he take HPHP as an example then? It should be obvious that there is not much interdependence with the other code.
Sounds to me like this: http://thedailywtf.com/Articles/Enterprise-Dependency-Big-Ba...