Hey do you know a lot of sites with such needs? Facebook is the first, and probably all sites with such needs I can count with fingers on my one hand. I don't think it's git issue - everyone use this system and all are happy using it. This is like a new feature, but not issue.
Facebook hit git performance issue on large repository
201–210 of 217 posts
Re: Facebook hit git performance issue on large repository
#202I can believe this working with a former facebook employee. They do not believe in separating or distilling anything into separate repos. Why the fuck would you want to have a 15GB repo? Ideally they should have many small, manageable repositories that are well tested and owned by a specific group/person/whatever. At least something small enough a single dev or team can get their head around. Sheesh.
And then each of those dev teams can spend 1/2 their time writing code other people in the company have already written or every team can spend 1/2 their time publishing and reading documentation about what has been written. There is no simple answer. There is only optimization for a particular problem-set you are trying to minimize.
> And then each of those dev teams can spend 1/2 their time writing code other people in the company have already written or every team can spend 1/2 their time publishing and reading documentation about what has been written.
I don't see what this has to do with a discussion of one repo vs multiple repos.
You think that in a multi repo world, the engineers aren't as aware of what code exists and where as they are in a single repo world? You think that code duplication and needing to read docs magically doesn't exist in a single repo world?
The number of repositories is just an organizational construct. Communication still must take place no matter what.
Re: Facebook hit git performance issue on large repository
#203Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…
Your comment was at the top so I continued to read expecting to find a bunch of ignorant group think about how git is awesome and Facebook is dumb, but that's not really what's going on down below. I don't know what facebook's use case is, so I have no idea if their repositories are optimally structured. However, I've used git on a very large repository and ran into some of the same performance issues that they did (…
It is intuitively obvious that it is better to be rich and healthy than poor and ill. Sadly, the reality choices are neither. And you can not just split a repo. Hearing some ideas for that case would have been interesting.
Solving a scaling problem by splitting it is, well, obvious.
And, yes, I also ran github on a couple of projects at $work and the issues are real, seen them.
So, if it hurts when I try to use git - the answer will be don't use git... But the conveniences are so tempting...
Re: Facebook hit git performance issue on large repository
#204Earlier quoted context omitted.
> Perforce is a great system, but it's showing it's age by now. Care to elaborate? Do you mean in terms of distributed -vs- centralised repos?
Yes partly. Doing lots of commits locally before pushing to others is definitely something I like. Another part of it is working disconnected -- with so many people coding on their laptops that's actually a pretty common use case. Also the lack of need to do sysadmin work on git/hg is really nice. I used to run the free Perforce server a long time ago for myself, but it was annoying to do the backups. With git or hg…
The most recent version of Perforce added streams which is their primary answer to git and hg. Easy creation, management, and switching between branches. I've only used this at home and not in a large scale environment yet, but it's promising.
Later this year they are adding p4 Sandbox which allows for disconnected work. When that is complete and working I'm honestly not sure what advantage git will have left other than being free.
Re: Facebook hit git performance issue on large repository
#205Earlier quoted context omitted.
True, but don't most places organize one git repo per project, rather than one for the entirety of the company's source code?
Most people like that use Perforce (e.g., Google). And no, they don't split into multiple repos, they might well have the entire company's source code in a single repository (code sharing is way easier this way).
That's a pretty terrible way to share code. Simple example: I work on a project, write some code. Turns out that code is useful for someone else, so they reach in and include it, which is easy since it's all the same repo and all delivered to the build system. Now I make a change and break some customer I didn't even know exists. Oops.
This is what package systems are for.
Re: Facebook hit git performance issue on large repository
#206Earlier quoted context omitted.
> They keep every project in a single repo, mystery solved. That's not true: > It is based on a growth model of two of our current repositories (I.e., it's not a perforce import). We already have some of the easily separable projects in separate repositories, like HPHP. If we could split our largest repos into multiple ones, that would help the scaling issue. However, the code in those repos is rather interdependent…
Why would he take HPHP as an example then? It should be obvious that there is not much interdependence with the other code. Sounds to me like this: http://thedailywtf.com/Articles/Enterprise-Dependency-Big-Ba...
Sounds to me like you're hell-bent on blaming Facebook instead of Git.
Re: Facebook hit git performance issue on large repository
#207Earlier quoted context omitted.
It seems eminently obvious to me that having basically a "change log" for a (part of a) filesystem is something that's valuable independent of your build system, revision control system, whatnot. At least that's what I'd like to see - it's functionality that's orthogonal to those tools.
Linux has been toying with a decent replacement for inotify for a while. Last time I looked it was called fanotify[1] and was still not merged. [1] https://lwn.net/Articles/339399/
Fanotify was merged but disabled in 2.6.36. It was enabled in 2.6.37.
Re: Facebook hit git performance issue on large repository
#208Earlier quoted context omitted.
Kernel/Android/Chrome/basically anything open-source is different. If the code is going to be open-sourced, it can't have dependencies on proprietary code anyway.
Right, so "literally the entire codebase for all their products" is incorrect. Thanks.
The open-source stuff is a rounding error. Think about all the Google products; Search, Google+, Gmail, Groups, Translate, Maps, Docs, Calendar, Checkout, Wallet, Voice, ... those are all in one repository. (Not to mention all the libraries and internal tools; those are all in there too.)
Re: Facebook hit git performance issue on large repository
#209Earlier quoted context omitted.
Right, so "literally the entire codebase for all their products" is incorrect. Thanks.
The open-source stuff is a rounding error. Think about all the Google products; Search, Google+, Gmail, Groups, Translate, Maps, Docs, Calendar, Checkout, Wallet, Voice, ... those are all in one repository. (Not to mention all the libraries and internal tools; those are all in there too.)
To be fair, Android and Chrome are pretty huge projects. I know the numbers (though I don't think I can share them outside of Google), and while they're nowhere close to being a big part of the total, they're also big enough to not be considered a rounding error.
Re: Facebook hit git performance issue on large repository
#210Earlier quoted context omitted.
Kernel/Android/Chrome/basically anything open-source is different. If the code is going to be open-sourced, it can't have dependencies on proprietary code anyway.
Right, so "literally the entire codebase for all their products" is incorrect. Thanks.
Ah,rachelbythebay, you caught him for being "technically incorrect". Which, depending on how you look at it, is either the best or worst kind of correctness.