Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook hit git performance issue on large repository

thread.gmane.org

Post reply on HN
Print view View original

141–150 of 217 posts

Re: Facebook hit git performance issue on large repository

#141

Post by groby_b »

jpdoctor wrote: post #70

$100B company, maybe they can afford to put some people onto solving this for the open software community (and put the solution into the open), especially since nobody else in the community seems to have this problem.

They are putting people on this. Thankfully, those people are smart enough to ask for help before blindly going off and doing their own thing.

Now, if it's going to end up OSS, that's a different question. (I'm not implying it's not - I'm saying that's a decision that could go either way)

Re: Facebook hit git performance issue on large repository

#142

Post by groby_b »

DannoHung wrote: post #126

Multiple people in this conversation section have asserted that code sharing is way easier when all the code is in a single repo , but from my understanding of sub-modules, it would be a fairly simple matter of setting up your pre/post-commit hooks to update submodules to a branch automatically and get useful company wide change atomicity (after all, changes should only propagate between teams/projects once they have…

If you limit change propagation, your changes won't propagate as fast. That goes for bugs and bug fixes.

I can certainly see why you would have the latter propagated instantaneously, or close to it.

There's also the point that if you don't propagate change to everybody at the same time, you'll have dozens of slightly different versions of those projects across your company. The question of submodules vs. large repo is not as easily decided as you think - there are large upsides (and downsides) to both approaches.

Re: Facebook hit git performance issue on large repository

#143

Post by yew »

EricBurnett wrote: post #101
mindcrime wrote: post #94

Earlier quoted context omitted.

Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstatio…

is anybody going to parse through all of that? Yes, in fact. We have some great tools that give us full search over our entire codebase (think Google Code Search), and you can add a dependency on a piece of code without needing to have it on your workstation already. The magic filesystem our build tools use knows where to get it and can do so on demand. Combined with good code location conventions, an overall attitud…

Knowing that somebody is taking the effort to get this sort of thing right is really, unspeakably awesome. Though I guess not really, given that I'm posting it. This sort of thing is one of the main reasons I read HN.

Hopefully the methodology will filter out into the wider world one day. . . Anyway, thank you for posting it!

Re: Facebook hit git performance issue on large repository

#144

Post by groby_b »

shabble wrote: post #138
groby_b wrote: post #135

Earlier quoted context omitted.

Which would also be tremendously useful for e.g. make.

There already exists tup : http://gittup.org/tup/ which does that sort of thing.

It seems eminently obvious to me that having basically a "change log" for a (part of a) filesystem is something that's valuable independent of your build system, revision control system, whatnot.

At least that's what I'd like to see - it's functionality that's orthogonal to those tools.

Re: Facebook hit git performance issue on large repository

#145

Post by cdibona »

ramanujan wrote: post #109

This looks like it could be of assistance: http://source.android.com/source/version-control.html Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow. Repo is not meant to replace Git, only to make it easier to work with Git in the context of Andr…

Basically, if you want to manage a large collection of git source repositories, you'll probably end up using Repo and Gerrit and piggybacking on the work of the android ecosystem (and beyond, gerrit is used all over the place now)

There really isn't another solution out there right now (at least not anything open source) for very large single repositories.

Re: Facebook hit git performance issue on large repository

#146

Post by forrestthewoods »

chubot wrote: post #111
forrestthewoods wrote: post #90

Earlier quoted context omitted.

It's called Perforce and anyone dealing with binary files has been using it for years.

And also paying Perforce fistfuls of cash in licensing fees. I hear that Perforce is a quite a small company, and the founder wrote the lion's share of the code a couple decades ago. I think they are probably on par with craigslist in profits per employee (i.e. much higher than Google or Facebook. Interestingly I think Facebook has about 1/10 the employees of Google with 1/10 the profits -- off the top of my head fee…

Perforce is quite expensive, yes. I don't understand the rest of your comments though. I'm not sure why company size, code author, or profit margins are relevant. Perforce is used by every major gaming studio, Pixar, Nvidia, and many more.

If I were to make a snarky comment it would be that Git is for poor people and Perforce is what you use when you grow up. That's not an even remotely reasonable statement, but it does have a teeny, tiny hint of truth to it. :)

Re: Facebook hit git performance issue on large repository

#147

Post by cheez »

groby_b wrote: post #144
shabble wrote: post #138

Earlier quoted context omitted.

There already exists tup : http://gittup.org/tup/ which does that sort of thing.

It seems eminently obvious to me that having basically a "change log" for a (part of a) filesystem is something that's valuable independent of your build system, revision control system, whatnot. At least that's what I'd like to see - it's functionality that's orthogonal to those tools.

Oh my god, that would be awesome at the FS level.

Re: Facebook hit git performance issue on large repository

#148

Post by pak »

nostrademons wrote: post #124
ghshephard wrote: post #22

Earlier quoted context omitted.

That C++ compiler is a single product (okay, you might have built a linker, and an assembler as well - say 3-5 products). In even medium enterprises (say, 500 employees, about 250 developers) you might have upwards of 35 different products, each of which with a 5-6 year active history. Enterprise source control can be ugly - particularly if you have non-text resources (Art, Firmware Binaries, tools) that need to be c…

There're costs and benefits both ways. AFAIK, Microsoft and Amazon both use the separate repositories model, and Google and Facebook use a single large repository. Most people I know that have worked at both of these styles prefer the Google/Facebook style. The biggest advantage of a single repository is pretty intangible - it's cultural. When anyone can change anything or can use any code, people feel like the whole…

Why is a single repo required for everybody to see all the code? Tools like gerrit and github can handle multiple repos and provide commit access for multiple repos among a large group of people. If it were my company, I would keep separate repos but allow read and merge requests for all employees. That keeps everybody involved in projects across the entire company, but also allows them to notice when individual projects get spaghettified and thereby deserving of some cleanup/breakup into components. A GB-scale codebase does not help smart, new employees grok what the hell they can contribute.

Re: Facebook hit git performance issue on large repository

#150

Post by karlshea »

jpdoctor wrote: post #133
rogerbinns wrote: post #122

Earlier quoted context omitted.

If you proposed a good solution I'm sure they'd be happy to provide time and money and open source the result. But most of the responses aren't even that there is a solution - they say to split the repository into smaller pieces and spend time and money internally having their internal developers deal with that. A good solution will benefit everyone who uses git. Codebases get larger over time. There is more forking…

> If you proposed a good solution I'm sure they'd be happy to provide time and money and open source the result. If they provided money, I'd provide the time in order to produce a good result. See the problem? More to the point: FB is all take and no give, as near as I can tell.

https://developers.facebook.com/opensource/

Looks like a fairly long list to me.

Post reply on HN