Facebook hit git performance issue on large repository • Hacker News Board

Re: Facebook hit git performance issue on large repository

#131

Post by nbm » Sat, Feb 04, 2012, 1:51 AM UTC

SomeCallMeTim wrote: post #118

nbm wrote: post #72

Earlier quoted context omitted.

I'm not sure this would work - if an operation needs you to stat() a file for every file in the repo (for example), whether it is 10k files in 1 repo or 1k files each in 10 repos will probably just as bad?

An operation in git makes you stat() each file in the current repo -- so things like check-ins and local operations would be done 100% in the current repo. Any time you were pulling the entire repo tree, it could be slow, yes. But assuming people are only working in one or a small number of repos at once, you can imagine a workflow that didn't involve nearly so many operations on the entire tree.

Ah, I see. Yes, that might make some of those operations better. Other operations that are common in our workflow might still need to look through the whole super-repo - for example getting a list of all changes (staged or unstaged) in the repo and generating a code review based on that.

(I almost habitually run "git status" whenever I've task switched away from code for even a few seconds to make sure I know exactly what I've done, which would have to look over the whole super-repo as well.)

Thankfully we're a while away from the times based on the synthetic test - it's not something I notice at all, but I probably write less code than most engineers here.

Re: Facebook hit git performance issue on large repository

#132

Post by akkartik » Sat, Feb 04, 2012, 1:55 AM UTC

lnguyen wrote: post #106

emelski wrote: post #102

Earlier quoted context omitted.

There's no need to spill internal processes and configurations. The fellow said he had a synthetic repo that he used to benchmark various operations. Surely whatever generated that test repo can scale it up or down to whatever size they like, so you can benchmark at various points and collect the data that would tell us if there is some horrible non-linear scaling going on under the covers.

Right now it sounds like he's just trying to see what the possible solutions for his issues are. If he can provide additional benchmarks, etc., great. But he's under no obligation to provide any more than he has. Once there's a solution, then maybe.

You're absolutely right, nobody is under any obligation to be non-disappointing.

He's soliciting opinions. I'm not sure how anybody can comment meaningfully based on the data he's given.

Re: Facebook hit git performance issue on large repository

#133

Post by jpdoctor » Sat, Feb 04, 2012, 1:59 AM UTC

rogerbinns wrote: post #122

jpdoctor wrote: post #70

$100B company, maybe they can afford to put some people onto solving this for the open software community (and put the solution into the open), especially since nobody else in the community seems to have this problem.

If you proposed a good solution I'm sure they'd be happy to provide time and money and open source the result. But most of the responses aren't even that there is a solution - they say to split the repository into smaller pieces and spend time and money internally having their internal developers deal with that. A good solution will benefit everyone who uses git. Codebases get larger over time. There is more forking…

> If you proposed a good solution I'm sure they'd be happy to provide time and money and open source the result.

If they provided money, I'd provide the time in order to produce a good result. See the problem?

More to the point: FB is all take and no give, as near as I can tell.

Re: Facebook hit git performance issue on large repository

#134

Post by jrockway » Sat, Feb 04, 2012, 2:08 AM UTC

Splines wrote: post #80

jrockway wrote: post #63

Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful. So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase,…

At Google, everything is in Perforce, but since I personally need only four or five projects from Perforce for my work, I mirror that to git and interact with git on a day-to-day basis. At MS we also use Perforce (aka Source Depot), and I've toyed with the idea of doing something similar. Have you found any guides for "gotchas" or care to share what you've learned going this route?

I used git-p4 at my last job, and the only thing that ever got weird was p4 branches. At Google we have an internal tool that's similar to git-p4, and it always works perfectly for me. Enough developers are using it such that most of the internal tools understand that a working copy could be a git repository instead of a p4 client.

So if you're planning on doing this at your own company, my advice is to write your own scripts that make whatever conventions you have automatic, and to move everyone over at the same time. That way, you won't be the weird one whose stuff is always broken.

I think most people got burned by cvs2svn and git-svn and think that using two version control systems at once is intrinsically broken. It's not. svn was just too weird to translate to or from. (People that skipped svn and went right from cvs to git had almost no problems, I'm told.)

Re: Facebook hit git performance issue on large repository

#135

Post by groby_b » Sat, Feb 04, 2012, 2:08 AM UTC

ori_b wrote: post #79

lincolnq wrote: post #55

Earlier quoted context omitted.

I had the same reaction as you. Stat'ing a million files is going to take a long time. Perforce doesn't have this problem because you explicitly check out files (p4 edit). (Perforce marks the whole tree read-only, as a reminder to edit the file before you save.) It seems like large-repo git could implement the same feature. You would just disable (or warn) for operations which require stat'ing the whole tree. Then th…

It seems to me that you could have a daemon that uses inotify to make operations O(changed) vs O(size).

Which would also be tremendously useful for e.g. make.

Re: Facebook hit git performance issue on large repository

#136

Post by nbm » Sat, Feb 04, 2012, 2:08 AM UTC

twinge wrote: post #130

SomeCallMeTim wrote: post #67

Earlier quoted context omitted.

None of what you mention here precludes breaking up the code into many smaller repositories, and then having them all linked together in one super-repository. Then tags at the super-repository level can record the exact state of all submodules. It's not about not checking the other modules out; you can make this the standard behavior, sure. Instead it's about having git manage reasonable sized blocks of the code base…

Three big problems with a split up codebase: 1) Instead of doing one large release every week (which facebook does: http://www.facebook.com/video/video.php?v=10100259101684977 ) you now have dozens or hundreds of smaller releases, a lot more heterogeneity to test for. 2) If you have inter-dependencies on modules you have to grapple with the "diamond dependency" problem. Say module A depends on module B and C, and sup…

Keep in mind that the talk in question is talking about the web site (and some other stuff) going into production, and as is mentioned in the talk, it is done more than once a week, and the whole shebang can be pushed in some number of minutes (I forget the amount mentioned).

Back-end services have their own release schedules and times, and obviously are made to be highly backward compatible so that they don't need to be done in lock-step with the front-end.

I think you're right about the "diamond dependency" model, but I think the merge-up and merge-down in Vista had more to do with having multiple independent branches in flight at the same time.

Re: Facebook hit git performance issue on large repository

#137

Post by rachelbythebay » Sat, Feb 04, 2012, 2:09 AM UTC

nostrademons wrote: post #125

rachelbythebay wrote: post #121

Earlier quoted context omitted.

The kernel? Android? Some other spooky stuff involving the pest control guy who's holding a big rubber mallet when you fail a unit test? Are you sure about that?

Kernel/Android/Chrome/basically anything open-source is different. If the code is going to be open-sourced, it can't have dependencies on proprietary code anyway.

Right, so "literally the entire codebase for all their products" is incorrect. Thanks.

Re: Facebook hit git performance issue on large repository

#138

Post by shabble » Sat, Feb 04, 2012, 2:16 AM UTC

groby_b wrote: post #135

ori_b wrote: post #79

Earlier quoted context omitted.

It seems to me that you could have a daemon that uses inotify to make operations O(changed) vs O(size).

Which would also be tremendously useful for e.g. make.

There already exists tup: http://gittup.org/tup/ which does that sort of thing.

Re: Facebook hit git performance issue on large repository

#139

Post by groby_b » Sat, Feb 04, 2012, 2:30 AM UTC

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

An inotify daemon could help, but it's not perfect: it needs a long time to warm up in the case of a reboot or crash

So does, presumably, the cache when you use lstat. (Let's scratch presumably. It does. Bonus points if you can't use Linux and use an OS that seems to chill its caches down as soon as possible. )

I hope I'm wrong, but the proper solution to this seems to be a custom file system - not only will it allow you to more easily obtain a "modified since" list of files, it also allows you to only get local files "on demand". (E.g. http://google-engtools.blogspot.com/2011/06/build-in-cloud-a...)

That still doesn't solve the data structure issues in git, but at least it takes some of the insane amount of I/O off the table.

I'm looking forward to see what you guys cook up :)

Re: Facebook hit git performance issue on large repository

#140

Post by gchpaco » Sat, Feb 04, 2012, 2:33 AM UTC

alttag wrote: post #108

to3m wrote: post #85

Earlier quoted context omitted.

I fear the market would be small. It is my guess (though I have no proof) that most places with particularly large repositories have lots of binary files in them. It's hard to get a 15GB repository if you just have text. This sort of thing suggests a centralized check-in/check-out model, because binary files are difficult to merge sensibly, and nobody wants to spend terabytes of hard drive space storing the repositor…

Yes, but why is that a show stopper? It's a small market filled only with people who typically have large fist-fulls of cash and are dependent on version control. It's a small market, but companies in it have the resources for a good solution.

Because those companies generally already pay the $$$ for Perforce (which has any number of deeply terrifying, shiny red candylike self destruct buttons and makes git's user interface look kind) which for all its other faults handles this specific user case extremely well.