Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

Post reply on HN
Print view View original

171–180 of 217 posts

Re: Facebook hit git performance issue on large repository

#171

Post by chubot »

forrestthewoods wrote: post #146
chubot wrote: post #111

Earlier quoted context omitted.

And also paying Perforce fistfuls of cash in licensing fees. I hear that Perforce is a quite a small company, and the founder wrote the lion's share of the code a couple decades ago. I think they are probably on par with craigslist in profits per employee (i.e. much higher than Google or Facebook. Interestingly I think Facebook has about 1/10 the employees of Google with 1/10 the profits -- off the top of my head fee…

Perforce is quite expensive, yes. I don't understand the rest of your comments though. I'm not sure why company size, code author, or profit margins are relevant. Perforce is used by every major gaming studio, Pixar, Nvidia, and many more. If I were to make a snarky comment it would be that Git is for poor people and Perforce is what you use when you grow up. That's not an even remotely reasonable statement, but it d…

I'm just pointing out the Perforce is making crazy profit, and somewhat ironically it's doing so more efficiently (I conjecture) than Facebook, which you are hearing a lot more about.

Perforce is a great system, but it's showing it's age by now. I think there is probably room for someone to make another product in the high end space and make boatloads of cash from big companies, but it's not easy.

Re: Facebook hit git performance issue on large repository

#172

Post by nfg »

chubot wrote: post #171
forrestthewoods wrote: post #146

Earlier quoted context omitted.

Perforce is quite expensive, yes. I don't understand the rest of your comments though. I'm not sure why company size, code author, or profit margins are relevant. Perforce is used by every major gaming studio, Pixar, Nvidia, and many more. If I were to make a snarky comment it would be that Git is for poor people and Perforce is what you use when you grow up. That's not an even remotely reasonable statement, but it d…

I'm just pointing out the Perforce is making crazy profit, and somewhat ironically it's doing so more efficiently (I conjecture) than Facebook, which you are hearing a lot more about. Perforce is a great system, but it's showing it's age by now. I think there is probably room for someone to make another product in the high end space and make boatloads of cash from big companies, but it's not easy.

> Perforce is a great system, but it's showing it's age by now.

Care to elaborate? Do you mean in terms of distributed -vs- centralised repos?

Re: Facebook hit git performance issue on large repository

#173

Post by philjohn »

joelhaasnoot wrote: post #62
jonursenbach wrote: post #52

Earlier quoted context omitted.

Not really relevant.

While it may not be, I do find it an every day battle to keep my PHP wel 'styled'. Sure, PHP is the first language I learnt, and I do use a framework, but sometimes a long method is easier in the short run than writing good model functions. And I have models, but mostly for the ORM and they're all completely interconnected. PHP makes me lazy, fast

That's pretty much the same in any language - if you don't have the discipline to keep your code in a good state in one, why would you suddenly gain that discipline in another?

Re: Facebook hit git performance issue on large repository

#174

Post by dochtman »

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

Heh, a familiar name working on this!

So for your first item, it seems like it should be possible to add a (mostly immutable) cache file doing the job of Mercurial's files field in changesets, right? I.e. for each commit, list the files changed. Should be more efficient than searching through trees/manifests for changed files, at least.

For large (in files) trees, it seems like there's no easy solution, except for developing some kind of subtree support. However, that's similar to just splitting up the repository (along the lines of hg subrepo support), in the sense that now you have no real verification that non-checked-out parts of the tree will work with the changes in the part you do have checked out.

Still, the inotify daemon seems like it could alleviate things a bunch; particularly if the repository is on a server anyway, i.e. it's not rebooted that often.

Re: Facebook hit git performance issue on large repository

#175

Post by malkia »

forrestthewoods wrote: post #146
chubot wrote: post #111

Earlier quoted context omitted.

And also paying Perforce fistfuls of cash in licensing fees. I hear that Perforce is a quite a small company, and the founder wrote the lion's share of the code a couple decades ago. I think they are probably on par with craigslist in profits per employee (i.e. much higher than Google or Facebook. Interestingly I think Facebook has about 1/10 the employees of Google with 1/10 the profits -- off the top of my head fee…

Perforce is quite expensive, yes. I don't understand the rest of your comments though. I'm not sure why company size, code author, or profit margins are relevant. Perforce is used by every major gaming studio, Pixar, Nvidia, and many more. If I were to make a snarky comment it would be that Git is for poor people and Perforce is what you use when you grow up. That's not an even remotely reasonable statement, but it d…

I love git, hg, svn, fossil but P4 is awesome.

I've used for 10+ at work, and it's easy to explain to anyone how to use it (from production to artist, and coders).

It has it's gotcha moments, but I haven't seen better. We normally go by 50-100GB depot, and then often whole branches of the game copied.

Re: Facebook hit git performance issue on large repository

#176

Post by exDM69 »

ramanujan wrote: post #109

This looks like it could be of assistance: http://source.android.com/source/version-control.html Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow. Repo is not meant to replace Git, only to make it easier to work with Git in the context of Andr…

> Looks like it's worth taking a serious look at this repo script, as it's been used in production for Android. Might allow splitting into multiple git repositories for performance while still retaining some of the benefits of a single repository.

Stay away from Repo and Gerrit. I use them at work, and they make my life miserable.

Repo was written years ago when Git did not have submodules, a feature where you can put repositories inside repositories. Git submodules is far superior to Repo, and allows you to e.g. bisect the history of many repositories.

I'm hoping that Google comes to it's senses and starts phasing out Repo in favor of Git submodules in Android development.

Re: Facebook hit git performance issue on large repository

#177

Post by DrCatbox »

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

You seem to know enough about this problem to solve it given enough time.

Why doesnt facebook solve this git on huge repos problem and put out a patch for others to see? Oh, right, you want somebody else to solve the problem for you, for free!

Re: Facebook hit git performance issue on large repository

#178

Post by patangay »

loeg wrote: post #56

I'm curious what their performance numbers look like if they host the .git repo on tmpfs -- 15GB isn't unreasonable on a beefy (24-32GB of ram) machine.

For what it's worth, Josh has been doing all his tests on machine with SSDs and 72GB of RAM.

Re: Facebook hit git performance issue on large repository

#179

Post by joelthelion »

exDM69 wrote: post #176
ramanujan wrote: post #109

This looks like it could be of assistance: http://source.android.com/source/version-control.html Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow. Repo is not meant to replace Git, only to make it easier to work with Git in the context of Andr…

> Looks like it's worth taking a serious look at this repo script, as it's been used in production for Android. Might allow splitting into multiple git repositories for performance while still retaining some of the benefits of a single repository. Stay away from Repo and Gerrit. I use them at work, and they make my life miserable. Repo was written years ago when Git did not have submodules, a feature where you can pu…

What about gerrit?

Re: Facebook hit git performance issue on large repository

#180

Post by joelthelion »

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

I see you've given the issue a lot of thought.

Sorry for asking the obvious, but do you really need huge amount of data to keep development productive? How often do you use history that is several years old? Could you not archive it?

Or is the sheer number of files the problem, even ignoring history?

This is not a "git is perfect, fix your workflow" post, but I'm genuinely interested in what you have to say. Also, it seems like making git faster is a increasingly difficult task, given the amount of effort that has already been put into it.

Post reply on HN