Facebook hit git performance issue on large repository • Hacker News Board

Re: Facebook hit git performance issue on large repository

#191

Post by Retric » Sat, Feb 04, 2012, 3:07 PM UTC

joeyh wrote: post #157

lincolnq wrote: post #55

Earlier quoted context omitted.

I had the same reaction as you. Stat'ing a million files is going to take a long time. Perforce doesn't have this problem because you explicitly check out files (p4 edit). (Perforce marks the whole tree read-only, as a reminder to edit the file before you save.) It seems like large-repo git could implement the same feature. You would just disable (or warn) for operations which require stat'ing the whole tree. Then th…

The git add problem is because .git/index is rewritten from scratch each time a new change is staged. With a 100 mb index file, that takes as long as it takes to write that much data to disk (cache). Much room for improvement here.

Writing 100mb to disk should take around 4 seconds on a HDD and less on a SSD.

Re: Facebook hit git performance issue on large repository

#192

Post by Retric » Sat, Feb 04, 2012, 3:13 PM UTC

lnguyen wrote: post #96

zobzu wrote: post #93

Earlier quoted context omitted.

if you were working for a truly open company, you could :)

I don't think Facebook is claiming to be. And as much as I'd like being truly open as an ideal, it falls apart when you're dealing with competition (not cooperation) and money. At best you try to keep things open enough.

I don't see how Facebook's build needs to be kept secret. It's a purely internal process and while they might lose something by giving details they can also gain if someone suggests improvements. That said, there are plenty of tings they need to keep secret. EX: Letting anyone export FB's full social graph would be really stupid.

Re: Facebook hit git performance issue on large repository

#193

Post by willvarfar » Sat, Feb 04, 2012, 3:30 PM UTC

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

Do you think adapting git to use, say, LevelDB and letting that do its job with incremental updates and maintaining a secondary index by path could help?

And that might dovetail nicely with an inotify daemon?

Re: Facebook hit git performance issue on large repository

#194

Post by [deleted] » Sat, Feb 04, 2012, 3:53 PM UTC

jrockway wrote: post #63

Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful. So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase,…

[deleted]

Re: Facebook hit git performance issue on large repository

#195

Post by tmhedberg » Sat, Feb 04, 2012, 4:21 PM UTC

pdw wrote: post #100

julian37 wrote: post #74

Somewhat off-topic, could somebody explain why echo 3 | tee /proc/sys/vm/drop_caches rather than just echo 3 > /proc/sys/vm/drop_caches Is it because the output to stdout lets you be extra sure that the right data was sent to the kernel? I'm just wondering if this is an idiom with a deeper meaning that I'm not aware of. EDIT: I'm guessing that when you run it in a script (without set -x), rather than on the command l…

Because you can echo 3 | sudo tee /proc/sys/vm/drop_caches but sudo echo 3 > /proc/sys/vm/drop_caches won't work.

You can also accomplish the same thing with:

    sudo sysctl -w vm.drop_caches=3

Re: Facebook hit git performance issue on large repository

#196

Post by rwmj » Sat, Feb 04, 2012, 6:31 PM UTC

groby_b wrote: post #144

shabble wrote: post #138

Earlier quoted context omitted.

There already exists tup : http://gittup.org/tup/ which does that sort of thing.

It seems eminently obvious to me that having basically a "change log" for a (part of a) filesystem is something that's valuable independent of your build system, revision control system, whatnot. At least that's what I'd like to see - it's functionality that's orthogonal to those tools.

Linux has been toying with a decent replacement for inotify for a while. Last time I looked it was called fanotify[1] and was still not merged.

[1] https://lwn.net/Articles/339399/

Re: Facebook hit git performance issue on large repository

#197

Post by jarjoura » Sat, Feb 04, 2012, 6:52 PM UTC

Arelius wrote: post #65

brown9-2 wrote: post #57

Earlier quoted context omitted.

True, but don't most places organize one git repo per project, rather than one for the entirety of the company's source code?

Most companies of this size don't use git. ;)

Heh, at Apple it's currently 50/50 svn/git.

Re: Facebook hit git performance issue on large repository

#198

Post by chubot » Sat, Feb 04, 2012, 7:02 PM UTC

nfg wrote: post #172

chubot wrote: post #171

Earlier quoted context omitted.

I'm just pointing out the Perforce is making crazy profit, and somewhat ironically it's doing so more efficiently (I conjecture) than Facebook, which you are hearing a lot more about. Perforce is a great system, but it's showing it's age by now. I think there is probably room for someone to make another product in the high end space and make boatloads of cash from big companies, but it's not easy.

> Perforce is a great system, but it's showing it's age by now. Care to elaborate? Do you mean in terms of distributed -vs- centralised repos?

Yes partly. Doing lots of commits locally before pushing to others is definitely something I like.

Another part of it is working disconnected -- with so many people coding on their laptops that's actually a pretty common use case.

Also the lack of need to do sysadmin work on git/hg is really nice. I used to run the free Perforce server a long time ago for myself, but it was annoying to do the backups. With git or hg you get whole-repository backups for free.

The "big repository with all dependencies model" has its drawbacks but it's interesting that facebook finds a lot of use for it, and that git is unsuitable for it. Perforce is probably still their best choice in that case.

Re: Facebook hit git performance issue on large repository

#199

Post by exDM69 » Sat, Feb 04, 2012, 7:50 PM UTC

cdibona wrote: post #145

ramanujan wrote: post #109

This looks like it could be of assistance: http://source.android.com/source/version-control.html Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow. Repo is not meant to replace Git, only to make it easier to work with Git in the context of Andr…

Basically, if you want to manage a large collection of git source repositories, you'll probably end up using Repo and Gerrit and piggybacking on the work of the android ecosystem (and beyond, gerrit is used all over the place now) There really isn't another solution out there right now (at least not anything open source) for very large single repositories.

> There really isn't another solution out there right now

What about Git submodules? They do fundamentally the same thing as Repo, but it's a built-in Git feature and not a bunch of scripts.

Repo can make your life very hard and you have to be a black belt Git ninja to understand what's going on when things don't go as you intended. Git submodules don't depend on having arbitrary GUID strings in your commit messages either (like Repo's Change-Id).

GitHub's reviews can handle Git submodules (but it's not free or open source). If someone knows any open source code review tools that can handle, please tell us.

Sorry for beating a dead horse, but I really want to save someone from fucking up (or at least re-centralizing) their workflow with repo scripts, when native git is better.

Re: Facebook hit git performance issue on large repository

#200

Post by redstone » Sat, Feb 04, 2012, 8:18 PM UTC

This is Joshua (who posted the original email). I'm glad to see so much interest in source control scalability. If there are others who have ever contemplated investing a bit of time to improving git, it'd be great to coordinate and see what makes sense to do - even if it turns out that the right answer is just to make the tools that manage multiple repos so good that it feels as easy as a single repo.