Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

41–50 of 245 posts

Re: Facebook's git repo is 54GB

#42
post #21

Earlier quoted context omitted.

The big .git directory is probably binary revisions? Is there any good way around that in git?

Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.

> Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo.

There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revisions count), broader (more files), more complex and probably full of binary stuff.

Re: Facebook's git repo is 54GB

#43
post #9

I bet most of that size is made up from the various dependencies Facebook probably has, though I'm still surprised it's that large. I expected the background worker things, like the facial recognition system for tagging people, and the video re-encoding libs, to be housed on separate repositories. I also wonder if that size includes a snapshot of a subset of Facebook's Graph, so that each developer has a "mini-facebo…

Having all code in a single repository increases developer productivity by lowering the barrier to change. You can make a single atomic commit in one repository as opposed to N commits in M repositories. This is much, much easier than dealing with subrepos, repo sync, etc.

Unified repos scales well up to a certain point before troubles arise. e.g. fully distributed VCS starts to break down when you have hundreds of MB and people with slow internet connections. Large projects like the Linux kernel and Firefox are beyond this point. You also have implementation details such as Git's repacks and garbage collection that introduce performance issues. Facebook is a magnitude past where troubles begin. The fact they control the workstations and can throw fast disks, CPU, memory, and 1 gbps+ links at the problem has bought them time.

Facebook made the determination that preserving a unified repository (and thus preserving developer productivity) was more important than dealing with the limitation of existing tools. So, they set out to improve one VCS system: Mercurial (https://code.facebook.com/posts/218678814984400/scaling-merc...). They are effectively leveraging the extensibility of Mercurial to turn it from a fully distributed VCS to one that supports shallow clones (remotefilelog extension) and can leverage filesystem watching primitives to make I/O operations fast (hgwatchman) and more. Unlike compiled tools (like Git), Facebook doesn't have to wait for upstream to accept possibly-controversial and difficult-to-land enhancements or maintain a forked Git distribution. They can write Mercurial extensions and monkeypatch the core of Mercurial (written in Python) to prove out ideas and they can upstream patches and extensions to benefit everybody. Mercurial is happily accepting their patches and every Mercurial user is better off because of Facebook.

Furthermore, Mercurial's extensibility makes it a perfect complement to a tailored and well-oiled development workflow. You can write Mercurial extensions that provide deep integration with existing tools and systems. See http://gregoryszorc.com/blog/2013/11/08/using-mercurial-to-q.... There are many compelling reasons why you would want to choose Mercurial over other solutions. Those reasons are even more compelling in corporate environments (such as Facebook) where the network effect of Git + GitHub (IMO the foremost reason to use Git) doesn't significantly factor into your decision.

Re: Facebook's git repo is 54GB

#45
post #41

Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?

That is a feature not a bug! Discoverability of code helps improve code quality and makes things less fragile.

Re: Facebook's git repo is 54GB

#46

Didn't they switch to Mercurial? https://code.facebook.com/posts/218678814984400/scaling-merc...

Somebody asked that on Twitter and the OP responded with:

>> At least according to the presentation by a Facebook engineer that I just watched, they're still on git. [0]

[0] https://twitter.com/feross/status/459335105853804544

Re: Facebook's git repo is 54GB

#47
I hope everyone realizes this is not 54GB of code, but in fact, is more likely a very public showing of very poor SCM management. They likely have tons of binaries in there, many many full codebase changes (whitespace, tabs, line endings, etc). Also not to mention how much dead code lives in there?

Re: Facebook's git repo is 54GB

#48
post #19

Earlier quoted context omitted.

They aim for a completely linear history. They may even have a policy of not allowing merge commits. It is described in various places on the internet. I like https://secure.phabricator.com/book/phabflavor/article/recom... because it and its sister articles on code review and revision control are terrific reads.

Dear everyone: you should be using Phabricator. It is Facebook's collected wisdom about software development expressed in software. It has improved my life substantially. The code review is better than Github's, and their linear, squashed commit philosophy has worked out much better than the way I used to do things.

That might be a silly question but it's not useful for just an individual in a team, correct? We use GitHub at work, but I wouldn't be able to try Phabricator on my own, right?

Re: Facebook's git repo is 54GB

#49
post #36
post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

I bet they check in (or have checked in at some point) 3rd party libraries, jar files, generated files, etc. I battle this every day at $dayjob and we have a multi GB subversion repository with a separate one for the 3rd party binaries. Svn handles this a bit better than the DVCSes, so just checking out the HEAD is smaller than the full history you get in git/hg, and you can clean up some crud a bit. It just lives on…

You can clone with a single revision in git. Still ends up being around 2X the size I believe since you have the objects stored for the single revision as well as the working tree.

Re: Facebook's git repo is 54GB

#50
post #41

Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?

Yes, and an intern would be subject to a code review before pushing to master - no way would they have write access to the "master" repo.
Post reply on HN