Facebook hit git performance issue on large repository • Hacker News Board

Re: Facebook hit git performance issue on large repository

#41

Post by sek » Fri, Feb 03, 2012, 9:54 PM UTC

mikeocool wrote: post #35

sek wrote: post #29

Earlier quoted context omitted.

Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.

No, literally the entire codebase for all of their products is in one Perforce repo. Ashish Kumar, manager of the Engineering Tools team, mentions it in this presentation: http://www.infoq.com/presentations/Development-at-Google

Very interesting, thank you.

Re: Facebook hit git performance issue on large repository

#42

Post by reid » Fri, Feb 03, 2012, 9:56 PM UTC

jlarocco wrote: post #27

wmf wrote: post #14

Earlier quoted context omitted.

Given that Facebook is compiled into a single 1 GB executable, a git repo with 1.3 M files doesn't really surprise me.

What? Do you have a reference for that?

Facebook Engineering posted a video about their build process in May 2011. Seek to 25:55 for the source: https://www.facebook.com/video/video.php?v=10100259101684977...

Re: Facebook hit git performance issue on large repository

#43

Post by earino » Fri, Feb 03, 2012, 9:56 PM UTC

If this was the crazy size of your git repo, why wouldn't you make a tool that took your git repo and versioned it? Keep it in repos that can all be performant, since most of the time you are working with "time local" information?

Re: Facebook hit git performance issue on large repository

#44

Post by nbm » Fri, Feb 03, 2012, 9:58 PM UTC

dpcx wrote: post #15

I don't want to imagine the actual kind of code that requires 1.3M files to run.

Keep in mind that your average repository doesn't only contain code that is compiled and executed (or interpreted), there is also documentation, static assets such as images (that may be processed), configuration, computed files (that may make sense to pre-compute once rather than compute on a hundred people's environments every build), and so forth.

Also, it doesn't only include the current file set - they include files that have been deleted, been split into modular files, been merged, been wholesale rewritten, put into a new hierarchy (some VCS systems handle this better than others).

(I work at Facebook, but not on the team looking into this stuff. I'm a happy user of their systems though. Keep in mind that the 1.3 million file repo is a synthetic test, not reality.)

Re: Facebook hit git performance issue on large repository

#45

Post by marginalboy » Fri, Feb 03, 2012, 9:59 PM UTC

losvedir wrote: post #28

Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases. So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebo…

It isn't surprising if Facebook has a large, highly coupled code base. Given their reputation for tight timelines and maverick-advocacy, I'm continually surprised the thing works at all.

Re: Facebook hit git performance issue on large repository

#46

Post by zargon » Fri, Feb 03, 2012, 10:00 PM UTC

losvedir wrote: post #28

Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases. So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebo…

The linux kernel has an order of magnitude fewer files than Facebook's test repository (25,000 in version 2.6.27, according to http://www.schoenitzer.de/lks/lks_en.html) and only 9 million lines of text.

This is on the largish side for a single project, but if Facebook likes to keep all their work in single repo then it isn't too difficult to go way beyond those stats. Think of keeping all GNU projects in a single repo.

Re: Facebook hit git performance issue on large repository

#47

Post by pwpwp » Fri, Feb 03, 2012, 10:01 PM UTC

Git was designed for the Linux kernel, and it's simply not big: a couple thousand files, broken up into directories of dozens or hundreds of files.

http://www.schoenitzer.de/lks/lks_en.html#new_files

Re: Facebook hit git performance issue on large repository

#48

Post by fragsworth » Fri, Feb 03, 2012, 10:05 PM UTC

sek wrote: post #19

http://thread.gmane.org/gmane.comp.version-control.git/18977... They keep every project in a single repo, mystery solved. Edit: > We already have some of the easily separable projects in separate repositories, like HPHP. Yeah, because it makes no sense, it's C++. They probably use for everything PHP i assume then. Is there no good build management tool for it?

> They keep every project in a single repo, mystery solved.

This kind of "Duh, look what you're doing" response isn't really justified.

Sure, splitting up your repository would make things faster, but having to maintain multiple repositories is a major headache for the end-users of git. If it's possible, why not fix its scalability so that you don't have to worry about it?

Re: Facebook hit git performance issue on large repository

#49

Post by joelhaasnoot » Fri, Feb 03, 2012, 10:07 PM UTC

cbs wrote: post #39

losvedir wrote: post #28

Huh, fascinating. git was initially created for the Linux kernel development, and I haven't heard of any issues there. Offhand I would have said, as a codebase, the Linux kernel would be larger and more complex than facebook, but I don't have a great sense of everything involved in both cases. So what's the story here: kernel developers put up with longer git times, the kernel is better organized, the scope of facebo…

From the sounds of it facebook has a really, really big ball of highly coupled code.

Sounds like PHP.

Oh wait, it is.

Re: Facebook hit git performance issue on large repository

#50

Post by ctz » Fri, Feb 03, 2012, 10:08 PM UTC

I'm surprised Facebook and all its peripheral development has that much source. I would expect something like 5-10 million lines of code, not ~100 million lines implied by the example.