Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

Post reply on HN
Print view View original

91–100 of 217 posts

Re: Facebook hit git performance issue on large repository

#91

Post by durin42 »

wmf wrote: post #66
loeg wrote: post #56

I'm curious what their performance numbers look like if they host the .git repo on tmpfs -- 15GB isn't unreasonable on a beefy (24-32GB of ram) machine.

Probably the same as the warm cache results, since that's basically what tmpfs is. I wonder if git does all that stat()ing serially or in parallel, though...

I don't have the link handy, but IIRC we did some experiments with that for Mercurial and found that stat() in parallel didn't really help much until you were using NFS or similarly slow network-type latency filesystems.

Re: Facebook hit git performance issue on large repository

#92

Post by lnguyen »

akkartik wrote: post #83
lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

You're right. I found the original email equally disappointing, though. It boils down to "We pushed the envelope on size, it's too slow, we'd like to speed it up." Well, duh. He uses the word 'scalability' early in the email, but shows no indication that he knows what it means. I'd love to hear if different operations slow down at different rates as the repo accumulates commits. Do they scale linearly, sublinearly, o…

It's intentionally vague but with enough details that if you're actually in a position to help, you'll recognize what's going on and actually directly contact to get more information.

You don't spill internal processes and configurations without some kind of disclosure agreements and certainly not in a public forum.

Re: Facebook hit git performance issue on large repository

#93

Post by zobzu »

lnguyen wrote: post #92
akkartik wrote: post #83

Earlier quoted context omitted.

You're right. I found the original email equally disappointing, though. It boils down to "We pushed the envelope on size, it's too slow, we'd like to speed it up." Well, duh. He uses the word 'scalability' early in the email, but shows no indication that he knows what it means. I'd love to hear if different operations slow down at different rates as the repo accumulates commits. Do they scale linearly, sublinearly, o…

It's intentionally vague but with enough details that if you're actually in a position to help, you'll recognize what's going on and actually directly contact to get more information. You don't spill internal processes and configurations without some kind of disclosure agreements and certainly not in a public forum.

if you were working for a truly open company, you could :)

Re: Facebook hit git performance issue on large repository

#94

Post by mindcrime »

EricBurnett wrote: post #82
dblock wrote: post #20

Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce. Google with all their source code in one Perforce repo. Facebook will be on perforce before we know it. The solution is an internal Github, not one giant proje…

Pointing at a big company and saying "they're doing it wrong" is easy enough to do, but you have to remember that every decision comes with tradeoffs. Take Google's codebase, since it's the one I know the best. A couple of the key decisions: * Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. * Build from head. We build everything from source, statically linked…

Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication.

I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstation?

And sure a compiled language would catch naming collisions on functions or whatever, but nothing stops somebody from creating a method

doQuickSort( ... )

and somebody else creating

quickSortFoo(...)

where they are semantically equivalent (or very nearly so).

It seems to me that the problem of duplicating code, because you don't know that a method already exists to do what you're trying to do, is the same problem regardless of how your tree is laid out; and is ultimately more of a documentation / process / discipline issue. But I'd be curious to hear the counter-argument to that...

Re: Facebook hit git performance issue on large repository

#95

Post by lnguyen »

fragsworth wrote: post #48
sek wrote: post #19

http://thread.gmane.org/gmane.comp.version-control.git/18977... They keep every project in a single repo, mystery solved. Edit: > We already have some of the easily separable projects in separate repositories, like HPHP. Yeah, because it makes no sense, it's C++. They probably use for everything PHP i assume then. Is there no good build management tool for it?

> They keep every project in a single repo, mystery solved. This kind of "Duh, look what you're doing" response isn't really justified. Sure, splitting up your repository would make things faster, but having to maintain multiple repositories is a major headache for the end-users of git. If it's possible, why not fix its scalability so that you don't have to worry about it?

I'm pretty sure that cloning a repository of that size can't be all that fun.

You tend to split repositories based on team responsibilities. I doubt that every developer needs access to update all million+ files.

What this comes down to is that they've made certain architecture decisions that ideally would be changed but it's not possible to do so at this time.

Re: Facebook hit git performance issue on large repository

#96

Post by lnguyen »

zobzu wrote: post #93
lnguyen wrote: post #92

Earlier quoted context omitted.

It's intentionally vague but with enough details that if you're actually in a position to help, you'll recognize what's going on and actually directly contact to get more information. You don't spill internal processes and configurations without some kind of disclosure agreements and certainly not in a public forum.

if you were working for a truly open company, you could :)

I don't think Facebook is claiming to be.

And as much as I'd like being truly open as an ideal, it falls apart when you're dealing with competition (not cooperation) and money. At best you try to keep things open enough.

Re: Facebook hit git performance issue on large repository

#97

Post by bos »

Facebook engineer here, working on this problem with Joshua.

What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful.

A few examples:

* There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file.

* Since git uses lstat to see if files have been changed, the sheer number of system calls on a large filesystem becomes an issue. If the dentry and inode caches aren't warm, you spend a ton of time waiting on disk I/O.

An inotify daemon could help, but it's not perfect: it needs a long time to warm up in the case of a reboot or crash. Also, inotify is an incredibly tricky interface to use efficiently and reliably. (I wrote the inotify support in Mercurial, FWIW.)

* The index is also a performance problem. On a big repo, it's 100MB+ in size (hence expensive to read), and the whole thing is rewritten from scratch any time it needs to be touched (e.g. a single file's stat entry goes stale).

None of these problems is insurmountable, but neither is any of them amenable to an easy solution. (And no, "split up the tree" is not an easy solution.)

Re: Facebook hit git performance issue on large repository

#98

Post by haberman »

cbs wrote: post #39
losvedir wrote: post #28

Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases. So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebo…

From the sounds of it facebook has a really, really big ball of highly coupled code.

Not necessarily.

In the open-source world, when you want to change an API, you have to either add the change as a new API (leaving the existing API intact) or break backward compatibility and maintain parallel versions, gradually migrating users off of the old version.

Both of these options are a huge pain, and have a direct cost (larger API surface or parallel maintenance/migration efforts). When your entire repo and all callers are in the same code-base, you have a much more attractive option: change the API and all callers in a single changelist. You've now cleaned up your API without incurring any of the costs of the two open-source options.

This is why it can be nice, even if you have a bunch of nicely structured components, to have all code in a single repository.

Re: Facebook hit git performance issue on large repository

#99

Post by akkartik »

lnguyen wrote: post #92
akkartik wrote: post #83

Earlier quoted context omitted.

You're right. I found the original email equally disappointing, though. It boils down to "We pushed the envelope on size, it's too slow, we'd like to speed it up." Well, duh. He uses the word 'scalability' early in the email, but shows no indication that he knows what it means. I'd love to hear if different operations slow down at different rates as the repo accumulates commits. Do they scale linearly, sublinearly, o…

It's intentionally vague but with enough details that if you're actually in a position to help, you'll recognize what's going on and actually directly contact to get more information. You don't spill internal processes and configurations without some kind of disclosure agreements and certainly not in a public forum.

I'm not asking for secrets here. I'm asking for some sign that he has a well-defined problem to solve.

How git performs as repo size grows to 15GB isn't hidden in a vault at facebook somewhere; I suspect they just haven't done anything more detailed than a superficial time measurement.

Re: Facebook hit git performance issue on large repository

#100

Post by pdw »

julian37 wrote: post #74

Somewhat off-topic, could somebody explain why echo 3 | tee /proc/sys/vm/drop_caches rather than just echo 3 > /proc/sys/vm/drop_caches Is it because the output to stdout lets you be extra sure that the right data was sent to the kernel? I'm just wondering if this is an idiom with a deeper meaning that I'm not aware of. EDIT: I'm guessing that when you run it in a script (without set -x), rather than on the command l…

Because you can

    echo 3 | sudo tee /proc/sys/vm/drop_caches
but

    sudo echo 3 > /proc/sys/vm/drop_caches
won't work.
Post reply on HN