Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

21–30 of 245 posts

Re: Facebook's git repo is 54GB

#21
post #3

Pay attention to the footnote: *8 GB plus 46 GB .git directory

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

Re: Facebook's git repo is 54GB

#22

Pay attention to the footnote: *8 GB plus 46 GB .git directory

The new depth option [1] on git clone as of 1.9.x should make the .git dir much more reasonable.

> --depth : Create a shallow clone with a history truncated to the specified number of revisions.

If the OS and applications that are creating and modifying files are all honest about the file dates, it should be possible to only scan dates instead of reading out every file. Or even use ionotify-like events to track what changed.

[1]: http://git-scm.com/docs/git-clone

EDIT: As sisk points out below --depth itself is not new, but as of 1.9 the limitations that previously came with shallow clones were lifted. Thanks sisk.

Re: Facebook's git repo is 54GB

#24
post #9

I bet most of that size is made up from the various dependencies Facebook probably has, though I'm still surprised it's that large. I expected the background worker things, like the facial recognition system for tagging people, and the video re-encoding libs, to be housed on separate repositories. I also wonder if that size includes a snapshot of a subset of Facebook's Graph, so that each developer has a "mini-facebo…

Surely a mini-graph would be something that would be loaded by a script, instead of being stored in the repo itself.

Re: Facebook's git repo is 54GB

#25

Earlier quoted context omitted.

Why's it bad to store source code on Git ?

Its not bad, is really nice, but Git has one problem, when you codebase is big, the process takes a long time, imagine git scanning those 8GB every time you do a commit, that is why Facebook was looking to port all their code to another VCS

I think it's worth making a distinction between the Git plumbing and the Git porcelain when talking about performance. The core functionality (the plumbing) is very fast regardless of repository size. The slowdowns people describe are almost always related to the porcelain commands, which are poorly optimized. Almost every porcelain-level command will cause Git to lstat() every file in your tree, as well as check for the presence of .gitignore files in all of the directories. It's very wasteful.

The fix for this is pretty simple: use filesystem watch hooks like inotify to update an lstat cache. I wrote something like this for an internal project and the speed difference was night and day. I remember reading that there had been progress on the inotify front on the git dev mailing list a few years ago, don't know what the current status is.

Re: Facebook's git repo is 54GB

#27
post #20

Earlier quoted context omitted.

Why's it bad to store source code on Git ?

[deleted]

I wouldn't say Facebook and Twitter are competitors. Their models are quite different.

- Facebook's complex privacy vs Twitter's binary "public or private"

- Facebook's real names vs Twitter's @usernames

- Facebook's freeform posting length vs Twitter's 140 character limit

Re: Facebook's git repo is 54GB

#28
post #19

Earlier quoted context omitted.

They aim for a completely linear history. They may even have a policy of not allowing merge commits. It is described in various places on the internet. I like https://secure.phabricator.com/book/phabflavor/article/recom... because it and its sister articles on code review and revision control are terrific reads.

Dear everyone: you should be using Phabricator. It is Facebook's collected wisdom about software development expressed in software. It has improved my life substantially. The code review is better than Github's, and their linear, squashed commit philosophy has worked out much better than the way I used to do things.

It looks pretty great. How does it compare to Atlassian products (if you have used any)?

Re: Facebook's git repo is 54GB

#30
post #9

I bet most of that size is made up from the various dependencies Facebook probably has, though I'm still surprised it's that large. I expected the background worker things, like the facial recognition system for tagging people, and the video re-encoding libs, to be housed on separate repositories. I also wonder if that size includes a snapshot of a subset of Facebook's Graph, so that each developer has a "mini-facebo…

Reminds me of a recent project we had. The code we wrote came in at about 150k after an npm and bower install it was at 192mb.
Post reply on HN