Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

Post reply on HN
Print view View original

151–160 of 217 posts

Re: Facebook hit git performance issue on large repository

#151

Post by kinofcain »

lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

Your comment was at the top so I continued to read expecting to find a bunch of ignorant group think about how git is awesome and Facebook is dumb, but that's not really what's going on down below.

I don't know what facebook's use case is, so I have no idea if their repositories are optimally structured. However, I've used git on a very large repository and ran into some of the same performance issues that they did (30+ seconds to run git status), so I don't think it's terribly hard to imagine they're in a similar situation.

What we did to solve it is exactly what you're excoriating the people below for suggesting: we split the repos and used other tools to manage multiple git repos, 'Repo' in some situations, git submodules in others.

However, we moved to that workflow mainly because it had a number of other advantages, not just because it made day-to-day git operations faster.

I hope git gets faster, some of the performance problems described are things we saw too, but things are always more complicated and I see nothing below that looks like the knee-jerk ignorant consensus you're describing.

Sometimes the answer to "it hurts when I do this" is "don't do that... because there's other ways to solve the same issue that work better for a number of other reasons and we haven't bothered fixing that particular one because most of the time the other way works better anyway."

Re: Facebook hit git performance issue on large repository

#152

Post by nostrademons »

pak wrote: post #148
nostrademons wrote: post #124

Earlier quoted context omitted.

There're costs and benefits both ways. AFAIK, Microsoft and Amazon both use the separate repositories model, and Google and Facebook use a single large repository. Most people I know that have worked at both of these styles prefer the Google/Facebook style. The biggest advantage of a single repository is pretty intangible - it's cultural. When anyone can change anything or can use any code, people feel like the whole…

Why is a single repo required for everybody to see all the code? Tools like gerrit and github can handle multiple repos and provide commit access for multiple repos among a large group of people. If it were my company, I would keep separate repos but allow read and merge requests for all employees. That keeps everybody involved in projects across the entire company, but also allows them to notice when individual proj…

It's not a matter of being able to see all the code, it's a matter of being able to see and modify all the code. It allows you to have a "just fix it" culture when people see something's broken, and it lets you write changes that span multiple projects without worrying about how your change will behave when it can't be committed atomically.

Re: Facebook hit git performance issue on large repository

#153

Post by __david__ »

cheez wrote: post #147
groby_b wrote: post #144

Earlier quoted context omitted.

It seems eminently obvious to me that having basically a "change log" for a (part of a) filesystem is something that's valuable independent of your build system, revision control system, whatnot. At least that's what I'd like to see - it's functionality that's orthogonal to those tools.

Oh my god, that would be awesome at the FS level.

Mac OS X's FSEvents API has something similar to that. When you create a FSEvent listener you can pass in an old event ID so the system can give you all the stuff that happened while you weren't listening [1]. Apple uses this for Time Machine (and I suspect Spotlight, too).

[1] https://developer.apple.com/library/mac/#documentation/Darwi...

Re: Facebook hit git performance issue on large repository

#154

Post by robfig »

lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

In violent agreement here.

Git and HG: 1. Require you to be sync'ed to tip before pushing. 2. Cannot selectively check out files.

The former means that in any reasonably sized team, you will be forced to sync 30 times a day, even if you are the only one editing your section of the source tree. The latter means that Joe who is checking in the libraries for (huge open source project) for some testing increases everyones repo by that much, forever, even if it's deleted later.

Needless to say, the universal response is that I'm doing it wrong. Perforce 4 life!

But seriously, it says that Google adopted Git for their repo --- does anyone know how they use it? I would expect them to want a linear history, but their teams are way too big to be able to have everyone sync'ed to tip to push...

Re: Facebook hit git performance issue on large repository

#155

Post by caf »

robfig wrote: post #154
lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

In violent agreement here. Git and HG: 1. Require you to be sync'ed to tip before pushing. 2. Cannot selectively check out files. The former means that in any reasonably sized team, you will be forced to sync 30 times a day, even if you are the only one editing your section of the source tree. The latter means that Joe who is checking in the libraries for (huge open source project) for some testing increases everyone…

Require you to be sync'ed to tip before pushing.

That's not the case. In fact, in the context of Linux kernel development, there's many emails on LKML where Linus is telling someone that they shouldn't be merging random-kernel-of-the-day into their development branch.

Re: Facebook hit git performance issue on large repository

#156

Post by caf »

groby_b wrote: post #139
bos wrote: post #97

Facebook engineer here, working on this problem with Joshua. What this comes down to is that git uses a lot of essentially O(n) data structures, and when n gets big, that can be painful. A few examples: * There's no secondary index from file or path name to commit hash. This is what slows down operations like "git blame": they have to search every commit to see if it touched a file. * Since git uses lstat to see if f…

An inotify daemon could help, but it's not perfect: it needs a long time to warm up in the case of a reboot or crash So does, presumably, the cache when you use lstat. (Let's scratch presumably. It does. Bonus points if you can't use Linux and use an OS that seems to chill its caches down as soon as possible. ) I hope I'm wrong, but the proper solution to this seems to be a custom file system - not only will it allow…

You might be able to do the "custom file system" as a pass-through FUSE filesystem.

Re: Facebook hit git performance issue on large repository

#157

Post by joeyh »

lincolnq wrote: post #55
lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

I had the same reaction as you. Stat'ing a million files is going to take a long time. Perforce doesn't have this problem because you explicitly check out files (p4 edit). (Perforce marks the whole tree read-only, as a reminder to edit the file before you save.) It seems like large-repo git could implement the same feature. You would just disable (or warn) for operations which require stat'ing the whole tree. Then th…

The git add problem is because .git/index is rewritten from scratch each time a new change is staged. With a 100 mb index file, that takes as long as it takes to write that much data to disk (cache). Much room for improvement here.

Re: Facebook hit git performance issue on large repository

#158

Post by ori_b »

__david__ wrote: post #153
cheez wrote: post #147

Earlier quoted context omitted.

Oh my god, that would be awesome at the FS level.

Mac OS X's FSEvents API has something similar to that. When you create a FSEvent listener you can pass in an old event ID so the system can give you all the stuff that happened while you weren't listening [1]. Apple uses this for Time Machine (and I suspect Spotlight, too). [1] https://developer.apple.com/library/mac/#documentation/Darwi...

What happens if a file is created and deleted multiple times? How does this avoid doing a complete walk of FS state and being O(size) itself?

Re: Facebook hit git performance issue on large repository

#159

Post by ori_b »

robfig wrote: post #154
lbrandy wrote: post #51

Wow. I was expecting an interesting discussion. I was disappointed. Apparently the consensus on hacker news is that there exists a repository size N above which the benefits of splitting the repo _always_ outweigh the negatives. And, if that wasn't absurd enough, we've decided that git can already handle N and the repository in question is clearly above N. And I guess all along we'll ignore the many massive organizat…

In violent agreement here. Git and HG: 1. Require you to be sync'ed to tip before pushing. 2. Cannot selectively check out files. The former means that in any reasonably sized team, you will be forced to sync 30 times a day, even if you are the only one editing your section of the source tree. The latter means that Joe who is checking in the libraries for (huge open source project) for some testing increases everyone…

Git is not used for their main repo. Git is used as a local cache for perforce where a branch roughly corresponds to a CL. Only subtrees of interest are checked out.

Re: Facebook hit git performance issue on large repository

#160

Post by joeyh »

to3m wrote: post #85
kenrik wrote: post #69

Earlier quoted context omitted.

With that in mind it seems like there is a market for a git replacement for these huge repos.

I fear the market would be small. It is my guess (though I have no proof) that most places with particularly large repositories have lots of binary files in them. It's hard to get a 15GB repository if you just have text. This sort of thing suggests a centralized check-in/check-out model, because binary files are difficult to merge sensibly, and nobody wants to spend terabytes of hard drive space storing the repositor…

The naive solution to binary files is a centralized model. But here's an alternate, fully distributed implementation: http://git-annex.branchable.com/

15 gb is a tiny, tiny repo. I have a 7 tb repo "here" (really, spread amoung various drives, servers, S3, etc). :)

Post reply on HN