[deleted]
[deleted]
Facebook hit git performance issue on large repository
31–40 of 217 posts
Re: Facebook hit git performance issue on large repository
#32the obvious answer, repeatedly mentioned in comments:
> factor into modules, one project per repo
where i work we have a project with clear module boundaries, but all in the same repo. we have an "app" and some dependencies including our platform/web framework. none of these are stable, they're all growing together. Commits on the app require changes in the platform, and in code review it is helpful to see things all together. Porting commits across different branches requires porting both the application change and the dependent platform changes. Often a client-specific branch will require severe one-off changes so the platform may diverge -- it is not practical for us (right now) to continually rebase client branches onto the latest platform.
this is just our experience, not facebook's, but lets face it: real life software isn't black and white, and discussion that doesn't acknowledge this isn't particularly helpful.
Re: Facebook hit git performance issue on large repository
#33Earlier quoted context omitted.
> They keep every project in a single repo, mystery solved. That's not true: > It is based on a growth model of two of our current repositories (I.e., it's not a perforce import). We already have some of the easily separable projects in separate repositories, like HPHP. If we could split our largest repos into multiple ones, that would help the scaling issue. However, the code in those repos is rather interdependent…
Why would he take HPHP as an example then? It should be obvious that there is not much interdependence with the other code. Sounds to me like this: http://thedailywtf.com/Articles/Enterprise-Dependency-Big-Ba...
He most likely used HPHP as an example because the message he was replying to also mentioned HPHP. At least that's how I interpreted it.
Re: Facebook hit git performance issue on large repository
#34Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce. Google with all their source code in one Perforce repo. Facebook will be on perforce before we know it. The solution is an internal Github, not one giant proje…
Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.
[deleted]
Re: Facebook hit git performance issue on large repository
#35Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce. Google with all their source code in one Perforce repo. Facebook will be on perforce before we know it. The solution is an internal Github, not one giant proje…
Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.
No, literally the entire codebase for all of their products is in one Perforce repo. Ashish Kumar, manager of the Engineering Tools team, mentions it in this presentation: http://www.infoq.com/presentations/Development-at-Google
Re: Facebook hit git performance issue on large repository
#36Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce. Google with all their source code in one Perforce repo. Facebook will be on perforce before we know it. The solution is an internal Github, not one giant proje…
Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.
[deleted]
Re: Facebook hit git performance issue on large repository
#37I can believe this working with a former facebook employee. They do not believe in separating or distilling anything into separate repos. Why the fuck would you want to have a 15GB repo?
Ideally they should have many small, manageable repositories that are well tested and owned by a specific group/person/whatever. At least something small enough a single dev or team can get their head around.
Sheesh.
Re: Facebook hit git performance issue on large repository
#38Earlier quoted context omitted.
Given that Facebook is compiled into a single 1 GB executable, a git repo with 1.3 M files doesn't really surprise me.
What? Do you have a reference for that?
For that and other info, check out the "Push" Tech Talk given last year by the Facebook release engineering team's leader, Chuck Rossi: https://www.facebook.com/video/video.php?v=10100259101684977
Re: Facebook hit git performance issue on large repository
#39Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases. So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebo…
From the sounds of it facebook has a really, really big ball of highly coupled code.
Re: Facebook hit git performance issue on large repository
#40Earlier quoted context omitted.
Given that Facebook is compiled into a single 1 GB executable, a git repo with 1.3 M files doesn't really surprise me.
What? Do you have a reference for that?
https://github.com/facebook/hiphop-php/wiki/Running-HipHop
It's how HPHP works.