Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

Post reply on HN
Print view View original

101–110 of 217 posts

Re: Facebook hit git performance issue on large repository

#101

Post by EricBurnett »

mindcrime wrote: post #94
EricBurnett wrote: post #82

Earlier quoted context omitted.

Pointing at a big company and saying "they're doing it wrong" is easy enough to do, but you have to remember that every decision comes with tradeoffs. Take Google's codebase, since it's the one I know the best. A couple of the key decisions: * Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. * Build from head. We build everything from source, statically linked…

Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstatio…

is anybody going to parse through all of that?

Yes, in fact. We have some great tools that give us full search over our entire codebase (think Google Code Search), and you can add a dependency on a piece of code without needing to have it on your workstation already. The magic filesystem our build tools use knows where to get it and can do so on demand. Combined with good code location conventions, an overall attitude that promotes reuse over rewrites* and mandatory code reviews where someone can suggest a better approach, we do a pretty good job. Not everything is eliminated, of course, but I'm pretty happy with the state of things.

To your example, we'd use the STL for most of our sorting needs, but if you were to want, say, case-insensitive string sorting, I can tell you where to find it (ASCII, UTF8, or other). If you want a random number, any RNG you could want is available. Most data structures you could name have been written and tested already. Libraries for controlling how your binaries dump core, command line flags are parsed, callbacks are invoked, etc etc are readily available. We really do reuse code as much as possible, and it's wonderful to have ready access to all of this whenever you could ask.

*At a method level, anyways...we're famous for writing ever more file systems ;).

Re: Facebook hit git performance issue on large repository

#102

Post by emelski »

lnguyen wrote: post #92
akkartik wrote: post #83

Earlier quoted context omitted.

You're right. I found the original email equally disappointing, though. It boils down to "We pushed the envelope on size, it's too slow, we'd like to speed it up." Well, duh. He uses the word 'scalability' early in the email, but shows no indication that he knows what it means. I'd love to hear if different operations slow down at different rates as the repo accumulates commits. Do they scale linearly, sublinearly, o…

It's intentionally vague but with enough details that if you're actually in a position to help, you'll recognize what's going on and actually directly contact to get more information. You don't spill internal processes and configurations without some kind of disclosure agreements and certainly not in a public forum.

There's no need to spill internal processes and configurations. The fellow said he had a synthetic repo that he used to benchmark various operations. Surely whatever generated that test repo can scale it up or down to whatever size they like, so you can benchmark at various points and collect the data that would tell us if there is some horrible non-linear scaling going on under the covers.

Re: Facebook hit git performance issue on large repository

#103

Post by jochu »

julian37 wrote: post #74

Somewhat off-topic, could somebody explain why echo 3 | tee /proc/sys/vm/drop_caches rather than just echo 3 > /proc/sys/vm/drop_caches Is it because the output to stdout lets you be extra sure that the right data was sent to the kernel? I'm just wondering if this is an idiom with a deeper meaning that I'm not aware of. EDIT: I'm guessing that when you run it in a script (without set -x), rather than on the command l…

Aside from reasons you mentioned, I can imagine it being because it easily allows one to add a sudo or being habit because of it. For example:

  echo 3 | sudo tee /proc/sys/vm/drop_caches
Will allow you to write as root and

  sudo echo 3 > /proc/sys/vm/drop_caches
Will be a permission error. It executes the echo as root and the write as the user

Re: Facebook hit git performance issue on large repository

#104

Post by moe »

nbm wrote: post #78
ctz wrote: post #50

I'm surprised Facebook and all its peripheral development has that much source. I would expect something like 5-10 million lines of code, not ~100 million lines implied by the example.

The example is synthetic, so don't worry too much about the implications. It is useful to keep in mind that Facebook isn't just the front-end (and isn't just code, also images, configuration, and so forth). Just talking about open source stuff, Facebook also generates code like Cassandra, Hive (data warehousing application), Phabricator (a code review and lifecycle tool), HipHop for PHP (the translator/compiler, the…

And all of that must live in a single repository... because?

Re: Facebook hit git performance issue on large repository

#105

Post by SoftwareMaven »

iamleppert wrote: post #37

I can believe this working with a former facebook employee. They do not believe in separating or distilling anything into separate repos. Why the fuck would you want to have a 15GB repo? Ideally they should have many small, manageable repositories that are well tested and owned by a specific group/person/whatever. At least something small enough a single dev or team can get their head around. Sheesh.

And then each of those dev teams can spend 1/2 their time writing code other people in the company have already written or every team can spend 1/2 their time publishing and reading documentation about what has been written.

There is no simple answer. There is only optimization for a particular problem-set you are trying to minimize.

Re: Facebook hit git performance issue on large repository

#106

Post by lnguyen »

emelski wrote: post #102
lnguyen wrote: post #92

Earlier quoted context omitted.

It's intentionally vague but with enough details that if you're actually in a position to help, you'll recognize what's going on and actually directly contact to get more information. You don't spill internal processes and configurations without some kind of disclosure agreements and certainly not in a public forum.

There's no need to spill internal processes and configurations. The fellow said he had a synthetic repo that he used to benchmark various operations. Surely whatever generated that test repo can scale it up or down to whatever size they like, so you can benchmark at various points and collect the data that would tell us if there is some horrible non-linear scaling going on under the covers.

Right now it sounds like he's just trying to see what the possible solutions for his issues are. If he can provide additional benchmarks, etc., great. But he's under no obligation to provide any more than he has. Once there's a solution, then maybe.

Re: Facebook hit git performance issue on large repository

#107

Post by julian37 »

pdw wrote: post #100
julian37 wrote: post #74

Somewhat off-topic, could somebody explain why echo 3 | tee /proc/sys/vm/drop_caches rather than just echo 3 > /proc/sys/vm/drop_caches Is it because the output to stdout lets you be extra sure that the right data was sent to the kernel? I'm just wondering if this is an idiom with a deeper meaning that I'm not aware of. EDIT: I'm guessing that when you run it in a script (without set -x), rather than on the command l…

Because you can echo 3 | sudo tee /proc/sys/vm/drop_caches but sudo echo 3 > /proc/sys/vm/drop_caches won't work.

Right, that's what I was missing. That makes sense, and come to think about it, is a very useful addition to my toolbox. Thanks pdw and jochu!

Re: Facebook hit git performance issue on large repository

#108

Post by alttag »

to3m wrote: post #85
kenrik wrote: post #69

Earlier quoted context omitted.

With that in mind it seems like there is a market for a git replacement for these huge repos.

I fear the market would be small. It is my guess (though I have no proof) that most places with particularly large repositories have lots of binary files in them. It's hard to get a 15GB repository if you just have text. This sort of thing suggests a centralized check-in/check-out model, because binary files are difficult to merge sensibly, and nobody wants to spend terabytes of hard drive space storing the repositor…

Yes, but why is that a show stopper? It's a small market filled only with people who typically have large fist-fulls of cash and are dependent on version control. It's a small market, but companies in it have the resources for a good solution.

Re: Facebook hit git performance issue on large repository

#109

Post by ramanujan »

This looks like it could be of assistance:

http://source.android.com/source/version-control.html

  Repo is a repository management tool that we built on top 
  of Git. Repo unifies the many Git repositories when 
  necessary, does the uploads to our revision control 
  system, and automates parts of the Android development 
  workflow. Repo is not meant to replace Git, only to make 
  it easier to work with Git in the context of Android. The 
  repo command is an executable Python script that you can 
  put anywhere in your path. In working with the Android 
  source files, you will use Repo for across-network 
  operations. For example, with a single Repo command you 
  can download files from multiple repositories into your 
  local working directory.
http://google-opensource.blogspot.com/2008/11/gerrit-and-rep...

  With approximately 8.5 million lines of code (not 
  including things like the Linux Kernel!), keeping this all 
  in one git tree would've been problematic for a few reasons:

  * We want to delineate access control based on location in the tree.
  * We want to be able to make some components replaceable at a later date.
  * We needed trivial overlays for OEMs and other projects who either aren't ready or aren't able to embrace open source.
  * We don't want our most technical people to spend their time as patch monkeys.

  The repo tool uses an XML-based manifest file describing 
  where the upstream repositories are, and how to merge them 
  into a single working checkout. repo will recurse across 
  all the git subtrees and handle uploads, pulls, and other 
  needed items. repo has built-in knowledge of topic 
  branches and makes working with them an essential part of 
  the workflow.
Looks like it's worth taking a serious look at this repo script, as it's been used in production for Android. Might allow splitting into multiple git repositories for performance while still retaining some of the benefits of a single repository.

Re: Facebook hit git performance issue on large repository

#110

Post by tonfa »

bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

Nice to see you're still in the DVCS business ;)
Post reply on HN