Facebook hit git performance issue on large repository • Hacker News Board

Re: Facebook hit git performance issue on large repository

#161

Post by wisty » Sat, Feb 04, 2012, 5:40 AM UTC

kinofcain wrote: post #151

lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

Your comment was at the top so I continued to read expecting to find a bunch of ignorant group think about how git is awesome and Facebook is dumb, but that's not really what's going on down below. I don't know what facebook's use case is, so I have no idea if their repositories are optimally structured. However, I've used git on a very large repository and ran into some of the same performance issues that they did (…

On a simliar note, I've heard there of people who would hit the limit on fortran files, so they put every variable into a function call to the next file, which itself contained one function and a function call to the next file after that (if necessary).

Making stuff modular is often a good idea.

Re: Facebook hit git performance issue on large repository

#162

Post by icehawk » Sat, Feb 04, 2012, 5:42 AM UTC

ori_b wrote: post #158

__david__ wrote: post #153

Earlier quoted context omitted.

Mac OS X's FSEvents API has something similar to that. When you create a FSEvent listener you can pass in an old event ID so the system can give you all the stuff that happened while you weren't listening [1]. Apple uses this for Time Machine (and I suspect Spotlight, too). [1] https://developer.apple.com/library/mac/#documentation/Darwi...

What happens if a file is created and deleted multiple times? How does this avoid doing a complete walk of FS state and being O(size) itself?

It doesn't. But it does mean you don't have to do it every time

This is a nice overview on FSevents

http://arstechnica.com/apple/reviews/2007/10/mac-os-x-10-5.a...

Re: Facebook hit git performance issue on large repository

#163

Post by patangay » Sat, Feb 04, 2012, 5:52 AM UTC

caf wrote: post #156

groby_b wrote: post #139

Earlier quoted context omitted.

An inotify daemon could help, but it's not perfect: it needs a long time to warm up in the case of a reboot or crash So does, presumably, the cache when you use lstat. (Let's scratch presumably. It does. Bonus points if you can't use Linux and use an OS that seems to chill its caches down as soon as possible. ) I hope I'm wrong, but the proper solution to this seems to be a custom file system - not only will it allow…

You might be able to do the "custom file system" as a pass-through FUSE filesystem.

Yea, a FUSE filesystem is also being considered as one of the possible solutions.

Re: Facebook hit git performance issue on large repository

#164

Post by charlieok » Sat, Feb 04, 2012, 5:53 AM UTC

I think it's a bad practice to keep a giant code base in one repo. Split the code base into purpose-specific modules, just as you would split any project into purpose-specific modules. In fact, those two things might well line up 1:1.

If a project depends on other projects, have it reference the other projects. Where appropriate, include exact version numbers and/or commit hashes. Gemfiles are good examples of this good practice at work.

Yes, git has submodules for this sort of thing, but after investigating that route, I decided against using git submodules. Use something independent of the VCS instead. Then git won't do weird or unexpected things when you switch branches. Also, you might want to mix in projects that use other version control systems. And really, why unnecessarily couple a project to its version control system?

If (when?), even after splitting a megaproject into manageable subprojects, these performance issues creep in, I'd certainly be interested in whatever improvements people are coming up with...

Re: Facebook hit git performance issue on large repository

#165

Post by teyc » Sat, Feb 04, 2012, 5:55 AM UTC

This is an interesting social AND a technical problem. The problem for FB is that it is all too easy for them to just fork git, create the necessary interfaces and then hope the git maintainers would accept it (they mightn't) or release it into the wild (and incur bad karma and wrath of OS developers who'd see this has schism or even heresy).

They've reached out to the developers on git, and I guess that's a first step.

Re: Facebook hit git performance issue on large repository

#166

Post by carey » Sat, Feb 04, 2012, 5:57 AM UTC

cheez wrote: post #147

groby_b wrote: post #144

Earlier quoted context omitted.

It seems eminently obvious to me that having basically a "change log" for a (part of a) filesystem is something that's valuable independent of your build system, revision control system, whatnot. At least that's what I'd like to see - it's functionality that's orthogonal to those tools.

Oh my god, that would be awesome at the FS level.

NTFS has this optionally in the "USN Change Journal"; see http://msdn.microsoft.com/en-us/library/aa363798.aspx. It's used by a few Microsoft features like indexing and file replication, but it's available to third party programs too.

Re: Facebook hit git performance issue on large repository

#167

Post by [deleted] » Sat, Feb 04, 2012, 6:20 AM UTC

haberman wrote: post #98

cbs wrote: post #39

Earlier quoted context omitted.

From the sounds of it facebook has a really, really big ball of highly coupled code.

Not necessarily. In the open-source world, when you want to change an API, you have to either add the change as a new API (leaving the existing API intact) or break backward compatibility and maintain parallel versions, gradually migrating users off of the old version. Both of these options are a huge pain, and have a direct cost (larger API surface or parallel maintenance/migration efforts). When your entire repo an…

[deleted]

Re: Facebook hit git performance issue on large repository

#168

Post by r15habh » Sat, Feb 04, 2012, 6:21 AM UTC

Does Facebook really believe that because they have the most users, they should also have the biggest git repo?

Amazon and Google have already solved this problem, and the solution is to reorganize things into smaller manageable packages.

Re: Facebook hit git performance issue on large repository

#169

Post by DannoHung » Sat, Feb 04, 2012, 6:35 AM UTC

groby_b wrote: post #142

DannoHung wrote: post #126

Multiple people in this conversation section have asserted that code sharing is way easier when all the code is in a single repo , but from my understanding of sub-modules, it would be a fairly simple matter of setting up your pre/post-commit hooks to update submodules to a branch automatically and get useful company wide change atomicity (after all, changes should only propagate between teams/projects once they have…

If you limit change propagation, your changes won't propagate as fast. That goes for bugs and bug fixes. I can certainly see why you would have the latter propagated instantaneously, or close to it. There's also the point that if you don't propagate change to everybody at the same time, you'll have dozens of slightly different versions of those projects across your company. The question of submodules vs. large repo i…

No, you're misunderstanding what I'm saying with regards to publishing stable changes.

Say you have two branches: master and next. Stable work goes in master, unstable work goes in next. When the code is ready for consumption, you merge it into master.

Anyone who is using a project has it setup as a submodule. They add post/pre-commit hooks to update all project submodules. These submodules pull from master.

This way, everyone will get all stable changes on all submodule projects at the time of the next change to their own project.

Re: Facebook hit git performance issue on large repository

#170

Post by LearnYouALisp » Sat, Feb 04, 2012, 6:49 AM UTC

wisty wrote: post #161

kinofcain wrote: post #151

Earlier quoted context omitted.

Your comment was at the top so I continued to read expecting to find a bunch of ignorant group think about how git is awesome and Facebook is dumb, but that's not really what's going on down below. I don't know what facebook's use case is, so I have no idea if their repositories are optimally structured. However, I've used git on a very large repository and ran into some of the same performance issues that they did (…

On a simliar note, I've heard there of people who would hit the limit on fortran files, so they put every variable into a function call to the next file, which itself contained one function and a function call to the next file after that (if necessary). Making stuff modular is often a good idea.

Cellular, modular, and interactive-odular!