Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

61–70 of 217 posts

Re: Facebook hit git performance issue on large repository

#61
post #19

http://thread.gmane.org/gmane.comp.version-control.git/18977... They keep every project in a single repo, mystery solved. Edit: > We already have some of the easily separable projects in separate repositories, like HPHP. Yeah, because it makes no sense, it's C++. They probably use for everything PHP i assume then. Is there no good build management tool for it?

> They keep every project in a single repo, mystery solved. This kind of "Duh, look what you're doing" response isn't really justified. Sure, splitting up your repository would make things faster, but having to maintain multiple repositories is a major headache for the end-users of git. If it's possible, why not fix its scalability so that you don't have to worry about it?

Git submodules make it really easy to split up large projects into smaller components.

Also, as Ævar writes in the first response, "there's only so much you can do about stat-ing 1.3 million files".

Re: Facebook hit git performance issue on large repository

#62

Earlier quoted context omitted.

Sounds like PHP. Oh wait, it is.

Not really relevant.

While it may not be, I do find it an every day battle to keep my PHP wel 'styled'. Sure, PHP is the first language I learnt, and I do use a framework, but sometimes a long method is easier in the short run than writing good model functions. And I have models, but mostly for the ORM and they're all completely interconnected. PHP makes me lazy, fast

Re: Facebook hit git performance issue on large repository

#63
Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful.

So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase, is different from the tool developers use for day-to-day work, which only needs to handle a small subset. At Google, everything is in Perforce, but since I personally need only four or five projects from Perforce for my work, I mirror that to git and interact with git on a day-to-day basis. This model seems to scale fairly well; Google has a big codebase with a lot of reuse, but all my git operations execute instantaneously.

Many projects can "shard" their code across repositories, but this is usually an unhappy compromise.

People always use the Linux kernel as an example of a big project, but even as open source projects go, it's pretty tiny. Compare the entire CPAN to Linux, for example. It's nice that I can update CPAN modules one at a time, but it would be nicer if I could fix a bug in my module and all modules that depend on it in one commit. But I can't, because CPAN is sharded across many different developers and repositories. This makes working on one module fast but working on a subset of modules impossible.

So really, Facebook is not being ridiculous here. Many companies have the same problem and decide not to handle it at all. Facebook realizes they want great developer tools and continuous integration across all their projects. And Git just doesn't work for that.

Re: Facebook hit git performance issue on large repository

#64

Earlier quoted context omitted.

Not really relevant.

While it may not be, I do find it an every day battle to keep my PHP wel 'styled'. Sure, PHP is the first language I learnt, and I do use a framework, but sometimes a long method is easier in the short run than writing good model functions. And I have models, but mostly for the ORM and they're all completely interconnected. PHP makes me lazy, fast

It's for this type of thinking that I've been looking at Rails recently.

Re: Facebook hit git performance issue on large repository

#65

Earlier quoted context omitted.

A lot of big companies have repos 10 or 100 times that size. With tens of millions of files, sometimes up to 100 gigs or more of data under source control.

True, but don't most places organize one git repo per project, rather than one for the entirety of the company's source code?

Most companies of this size don't use git. ;)

Re: Facebook hit git performance issue on large repository

#66
post #56

I'm curious what their performance numbers look like if they host the .git repo on tmpfs -- 15GB isn't unreasonable on a beefy (24-32GB of ram) machine.

Probably the same as the warm cache results, since that's basically what tmpfs is. I wonder if git does all that stat()ing serially or in parallel, though...

Re: Facebook hit git performance issue on large repository

#67
post #59

Earlier quoted context omitted.

It isn't surprising if Facebook has a large, highly coupled code base. Given their reputation for tight timelines and maverick-advocacy, I'm continually surprised the thing works at all.

I wouldn't say that a large repository implies that the code is highly-coupled. There are advantages for keeping certain code together in a single repo. Being able to easily discover users of functions of a library, being able to "transactionally" commit an update to a library (or set of libraries) and the code that uses it, being able to do code review over changes of code in various places, being able to discover i…

None of what you mention here precludes breaking up the code into many smaller repositories, and then having them all linked together in one super-repository.

Then tags at the super-repository level can record the exact state of all submodules.

It's not about not checking the other modules out; you can make this the standard behavior, sure. Instead it's about having git manage reasonable sized blocks of the code base.

Re: Facebook hit git performance issue on large repository

#68

Earlier quoted context omitted.

A lot of big companies have repos 10 or 100 times that size. With tens of millions of files, sometimes up to 100 gigs or more of data under source control.

True, but don't most places organize one git repo per project, rather than one for the entirety of the company's source code?

Most people like that use Perforce (e.g., Google).

And no, they don't split into multiple repos, they might well have the entire company's source code in a single repository (code sharing is way easier this way).

Re: Facebook hit git performance issue on large repository

#69
post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

With that in mind it seems like there is a market for a git replacement for these huge repos.
Post reply on HN