Facebook's git repo is 54GB
41–50 of 245 posts
Re: Facebook's git repo is 54GB
#42Earlier quoted context omitted.
The big .git directory is probably binary revisions? Is there any good way around that in git?
Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.
There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revisions count), broader (more files), more complex and probably full of binary stuff.
Re: Facebook's git repo is 54GB
#43I bet most of that size is made up from the various dependencies Facebook probably has, though I'm still surprised it's that large. I expected the background worker things, like the facial recognition system for tagging people, and the video re-encoding libs, to be housed on separate repositories. I also wonder if that size includes a snapshot of a subset of Facebook's Graph, so that each developer has a "mini-facebo…
Unified repos scales well up to a certain point before troubles arise. e.g. fully distributed VCS starts to break down when you have hundreds of MB and people with slow internet connections. Large projects like the Linux kernel and Firefox are beyond this point. You also have implementation details such as Git's repacks and garbage collection that introduce performance issues. Facebook is a magnitude past where troubles begin. The fact they control the workstations and can throw fast disks, CPU, memory, and 1 gbps+ links at the problem has bought them time.
Facebook made the determination that preserving a unified repository (and thus preserving developer productivity) was more important than dealing with the limitation of existing tools. So, they set out to improve one VCS system: Mercurial (https://code.facebook.com/posts/218678814984400/scaling-merc...). They are effectively leveraging the extensibility of Mercurial to turn it from a fully distributed VCS to one that supports shallow clones (remotefilelog extension) and can leverage filesystem watching primitives to make I/O operations fast (hgwatchman) and more. Unlike compiled tools (like Git), Facebook doesn't have to wait for upstream to accept possibly-controversial and difficult-to-land enhancements or maintain a forked Git distribution. They can write Mercurial extensions and monkeypatch the core of Mercurial (written in Python) to prove out ideas and they can upstream patches and extensions to benefit everybody. Mercurial is happily accepting their patches and every Mercurial user is better off because of Facebook.
Furthermore, Mercurial's extensibility makes it a perfect complement to a tailored and well-oiled development workflow. You can write Mercurial extensions that provide deep integration with existing tools and systems. See http://gregoryszorc.com/blog/2013/11/08/using-mercurial-to-q.... There are many compelling reasons why you would want to choose Mercurial over other solutions. Those reasons are even more compelling in corporate environments (such as Facebook) where the network effect of Git + GitHub (IMO the foremost reason to use Git) doesn't significantly factor into your decision.
Re: Facebook's git repo is 54GB
#44Re: Facebook's git repo is 54GB
#45Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?
Re: Facebook's git repo is 54GB
#46Didn't they switch to Mercurial? https://code.facebook.com/posts/218678814984400/scaling-merc...
>> At least according to the presentation by a Facebook engineer that I just watched, they're still on git. [0]
Re: Facebook's git repo is 54GB
#47Re: Facebook's git repo is 54GB
#48Earlier quoted context omitted.
They aim for a completely linear history. They may even have a policy of not allowing merge commits. It is described in various places on the internet. I like https://secure.phabricator.com/book/phabflavor/article/recom... because it and its sister articles on code review and revision control are terrific reads.
Dear everyone: you should be using Phabricator. It is Facebook's collected wisdom about software development expressed in software. It has improved my life substantially. The code review is better than Github's, and their linear, squashed commit philosophy has worked out much better than the way I used to do things.
Re: Facebook's git repo is 54GB
#49Earlier quoted context omitted.
8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.
I bet they check in (or have checked in at some point) 3rd party libraries, jar files, generated files, etc. I battle this every day at $dayjob and we have a multi GB subversion repository with a separate one for the 3rd party binaries. Svn handles this a bit better than the DVCSes, so just checking out the HEAD is smaller than the full history you get in git/hg, and you can clean up some crud a bit. It just lives on…
Re: Facebook's git repo is 54GB
#50Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?