Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

Post reply on HN
Print view View original

71–80 of 217 posts

Re: Facebook hit git performance issue on large repository

#71

Post by sek »

dustingetz wrote: post #32

the obvious answer, repeatedly mentioned in comments: > factor into modules, one project per repo where i work we have a project with clear module boundaries, but all in the same repo. we have an "app" and some dependencies including our platform/web framework. none of these are stable, they're all growing together. Commits on the app require changes in the platform, and in code review it is helpful to see things all…

This is what git submodules are for, but when they can't use them they don't have clear module boundaries.

Re: Facebook hit git performance issue on large repository

#72

Post by nbm »

SomeCallMeTim wrote: post #67
nbm wrote: post #59

Earlier quoted context omitted.

I wouldn't say that a large repository implies that the code is highly-coupled. There are advantages for keeping certain code together in a single repo. Being able to easily discover users of functions of a library, being able to "transactionally" commit an update to a library (or set of libraries) and the code that uses it, being able to do code review over changes of code in various places, being able to discover i…

None of what you mention here precludes breaking up the code into many smaller repositories, and then having them all linked together in one super-repository. Then tags at the super-repository level can record the exact state of all submodules. It's not about not checking the other modules out; you can make this the standard behavior, sure. Instead it's about having git manage reasonable sized blocks of the code base…

I'm not sure this would work - if an operation needs you to stat() a file for every file in the repo (for example), whether it is 10k files in 1 repo or 1k files each in 10 repos will probably just as bad?

Re: Facebook hit git performance issue on large repository

#73

Post by gokhan »

Large repos bring their own problems, and results in some design decisions accordingly. For example, Visual Studio itself is 5M+ files and this affected some of the the initial design decisions (Server side workspaces, for this example) when developing TFS 2005 (the first version) [1]. That decision suits MS but not the small to medium clients well. So they're now alternating that design with client side workspaces.

It's not wise to offer Facebook to split the repository. Looks like it's time to improve the tool.

[1] http://blogs.msdn.com/b/bharry/archive/2011/08/02/version-co...

Re: Facebook hit git performance issue on large repository

#74

Post by julian37 »

Somewhat off-topic, could somebody explain why

  echo 3 | tee /proc/sys/vm/drop_caches
rather than just

  echo 3 > /proc/sys/vm/drop_caches
Is it because the output to stdout lets you be extra sure that the right data was sent to the kernel?

I'm just wondering if this is an idiom with a deeper meaning that I'm not aware of.

EDIT: I'm guessing that when you run it in a script (without set -x), rather than on the command line, you can see in the log what it is you sent?

Re: Facebook hit git performance issue on large repository

#75

Post by djtriptych »

jamespo wrote: post #54
lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

I would have liked this comment better if it came up with some solutions itself, although maybe it's not easy to solve?

That's the problem. It's NOT an easy problem to solve.

A lot of posts on hn describing some problem elicit "Why, that's no problem at all!" responses or "That's the wrong problem to think about" responses.

Honestly that mindset is often really useful in programming, but when we get a problem that doesn't have a shortcut and is relevent, conversation goes to shit. Because I guess that's when programmers normally go into a hole and brute-force brain it out.

How to use mass comms to talk about a difficult open problem is, I suppose, itself an open problem.

Re: Facebook hit git performance issue on large repository

#76

Post by 0x0 »

nbm wrote: post #44
dpcx wrote: post #15

I don't want to imagine the actual kind of code that requires 1.3M files to run.

Keep in mind that your average repository doesn't only contain code that is compiled and executed (or interpreted), there is also documentation, static assets such as images (that may be processed), configuration, computed files (that may make sense to pre-compute once rather than compute on a hundred people's environments every build), and so forth. Also, it doesn't only include the current file set - they include f…

Also, it doesn't only include the current file set - they include files that have been deleted, been split into modular files, been merged, been wholesale rewritten, put into a new hierarchy (some VCS systems handle this better than others).

The follow-up email still mentions a working directory of 9.5gb. I cannot fathom working on a code repository consisting of 9.5gb of text. There must be something else going on here, even considering any peripheral projects like the iOS and android apps, etc.

(edit: if there are huge generated files intermingled with code, shouldn't those be hosted on a "pre-generated cache" web server instead of git, for example?)

Re: Facebook hit git performance issue on large repository

#78

Post by nbm »

ctz wrote: post #50

I'm surprised Facebook and all its peripheral development has that much source. I would expect something like 5-10 million lines of code, not ~100 million lines implied by the example.

The example is synthetic, so don't worry too much about the implications.

It is useful to keep in mind that Facebook isn't just the front-end (and isn't just code, also images, configuration, and so forth).

Just talking about open source stuff, Facebook also generates code like Cassandra, Hive (data warehousing application), Phabricator (a code review and lifecycle tool), HipHop for PHP (the translator/compiler, the interpreter, and the virtual machine), FlashCache (a kernel driver), Thrift, Scribe, and so forth.

We also have had to build applications to support our operations, so think about what sort of effort goes into building scalable monitoring, configuration management, automatic remediation, logging infrastructure, and so forth.

I don't know the actual lines of code across it all, and wouldn't mention it if I did, but people often underestimate the scale here.

Re: Facebook hit git performance issue on large repository

#79

Post by ori_b »

lincolnq wrote: post #55
lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

I had the same reaction as you. Stat'ing a million files is going to take a long time. Perforce doesn't have this problem because you explicitly check out files (p4 edit). (Perforce marks the whole tree read-only, as a reminder to edit the file before you save.) It seems like large-repo git could implement the same feature. You would just disable (or warn) for operations which require stat'ing the whole tree. Then th…

It seems to me that you could have a daemon that uses inotify to make operations O(changed) vs O(size).

Re: Facebook hit git performance issue on large repository

#80

Post by Splines »

jrockway wrote: post #63

Yes, it's well known that big companies with big continuously integrated codebases don't manage the entire codebase with Git. It's slow, and splitting repositories means you can't have company-wide atomic commits. It's convenient to have a bunch of separate projects that share no state or code, but also wasteful. So often, the tool used to manage the central repository, which needs to cleanly handle a large codebase,…

At Google, everything is in Perforce, but since I personally need only four or five projects from Perforce for my work, I mirror that to git and interact with git on a day-to-day basis.

At MS we also use Perforce (aka Source Depot), and I've toyed with the idea of doing something similar. Have you found any guides for "gotchas" or care to share what you've learned going this route?

Post reply on HN