Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

31–40 of 245 posts

Re: Facebook's git repo is 54GB

#31
Someone recently told me that Facebook had a torrent file that went around the company that people could use to download the entire codebase using a BitTorrent client. Is there any truth in this?

I mean, the same guy that told me this, also said that the codebase size was about 50 times less than the one reported in this slide, so it may all be pure speculation.

Re: Facebook's git repo is 54GB

#32
post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

Exactly, there's no way they wrote 8GB of code.

I think they did, have heard similar figures from them before, all deployed as one file. Macroservices...

Re: Facebook's git repo is 54GB

#33
post #23

Is there a reason why they keep everything in the same repo? Can’t you just split the code across multiple smaller repos?

It becoms a lot harder to keep everything in sync, especially if internal interfaces change frequently. At facebook scale though it's probably a good idea to defined boundaries between areas in the application better.

Re: Facebook's git repo is 54GB

#34

Earlier quoted context omitted.

Why's it bad to store source code on Git ?

Its not bad, is really nice, but Git has one problem, when you codebase is big, the process takes a long time, imagine git scanning those 8GB every time you do a commit, that is why Facebook was looking to port all their code to another VCS

" when you codebase is big, the process takes a long time"

From my experience, we can bet git is the one that takes the least amount of time

Not hitting the network to check which files changed, for a start

Re: Facebook's git repo is 54GB

#35
post #21
post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.

Re: Facebook's git repo is 54GB

#36
post #3

Pay attention to the footnote: *8 GB plus 46 GB .git directory

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

I bet they check in (or have checked in at some point) 3rd party libraries, jar files, generated files, etc. I battle this every day at $dayjob and we have a multi GB subversion repository with a separate one for the 3rd party binaries. Svn handles this a bit better than the DVCSes, so just checking out the HEAD is smaller than the full history you get in git/hg, and you can clean up some crud a bit. It just lives on the central server, not in everyone's working copy.

Re: Facebook's git repo is 54GB

#38

Someone recently told me that Facebook had a torrent file that went around the company that people could use to download the entire codebase using a BitTorrent client. Is there any truth in this? I mean, the same guy that told me this, also said that the codebase size was about 50 times less than the one reported in this slide, so it may all be pure speculation.

Facebook uses BitTorrent to deploy their binaries to their many servers.

So using torrents isn't foreign to them.

Re: Facebook's git repo is 54GB

#39
The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about.

I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream modules because it's not their problem anymore. Software in fast moving companies doesn't work like that. There are always subtle behavior dependancies (re: one module depends on a bug in another module either by mistake or intentionally). I just prefer having all code and tests of all modules in one place.

Re: Facebook's git repo is 54GB

#40
post #8

I thought I had read an article about facebook switching to perforce due to their really large git repo. Were they at least thinking about it? A quick google comes up with nothing but I could have SWORN I read that.

You are thinking about Mercurial, not Perforce: https://code.facebook.com/posts/218678814984400/scaling-merc...
Post reply on HN