Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

161–170 of 245 posts

Re: Facebook's git repo is 54GB

#161
post #154
post #102

Earlier quoted context omitted.

Yeah, doing it this way they can never make API incompatible changes without also fixing everything downstream... which effectively means once a library is popular enough it is locked at it's current API forever.

Or you bump an API version. And the fixes are gradual everywhere.

The parent was advocating not versioning.

Re: Facebook's git repo is 54GB

#162
post #108
post #59

Earlier quoted context omitted.

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

They built their own build system because once you are dealing with top engineers NIH sets in quickly and you write your own everything.

To be fair once you are over 1000 engineers, a 1% improvement in productivity is worth a lot of development time.

Re: Facebook's git repo is 54GB

#163
post #21
post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

FWIW: Every time you modify a file in git, it adds a completely new copy of the file. 100mb text file, 100mb binary file - makes no difference. Modify one line, it's a new 100mb entry in your git repo.

Re: Facebook's git repo is 54GB

#164
post #69

Earlier quoted context omitted.

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

I wonder, if this is the way a majority of big businesses do things, how come we don't see more leaks of entire codebases? It'd be trivial to put something up on TPB and just share all the code, but I don't see things like that happening. I also doubt that every single employee with access to the code has the moral standards not to do this. There must be something else keeping them from doing it.

I'd say the combination of some moral standards, the fear of getting caught and - first of all - little incentive to leak should be sufficient, no?

Re: Facebook's git repo is 54GB

#165
post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

8GB would be at least 100 million lines of code (upper bound with 80 characters per each line). For comparison Linux has 15+ million lines of code, PHP 2+M.

PHP's repo is around 500M. But I'd say it is probably tens to a hundred times smaller than Facebook should be, especially if you count non-public stuff they must have there. So comes out about right.

Re: Facebook's git repo is 54GB

#166
post #155

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

Most big tech companies use a service-oriented architecture. The website is composed of many small services which communicate with each other over HTTP or RPC protocols. Each service has its own version control repo and is maintained by a different team. That's generally the best way of scaling up.

Facebook's architecture, for at least the core web app, was a blob of PHP for ages.

They might have since modularized and cleaned it up but it seems unlikely they'd fully SOA-ize the Facebook web app.

Re: Facebook's git repo is 54GB

#167
post #21

Earlier quoted context omitted.

The big .git directory is probably binary revisions? Is there any good way around that in git?

I'm a bit confused. Whenever I've used git on my projects, I'd make sure the binaries were excluded, using .gitignore Don't other people do that, too? What's the benefit of having binaries stored? I've never needed that; I've never worked on any huge projects, so I might be missing something crucial.

If there is a small number of rarely changing binaries (like icons, tool configs, etc.) then it may not be worth it to move them. Also if space is much cheaper than tool complexity and build time.

Re: Facebook's git repo is 54GB

#169

Someone recently told me that Facebook had a torrent file that went around the company that people could use to download the entire codebase using a BitTorrent client. Is there any truth in this? I mean, the same guy that told me this, also said that the codebase size was about 50 times less than the one reported in this slide, so it may all be pure speculation.

That sounds entirely believable. At $PREVIOUS_JOB, the Puppet git repository was large enough (several GB) that cloning it was painful, and new starters were handed a pruned repository via the local network so that they could get something done today.

Re: Facebook's git repo is 54GB

#170
post #21

Earlier quoted context omitted.

The big .git directory is probably binary revisions? Is there any good way around that in git?

FWIW: Every time you modify a file in git, it adds a completely new copy of the file. 100mb text file, 100mb binary file - makes no difference. Modify one line, it's a new 100mb entry in your git repo.

No, git has delta compression. It only saves the changes.
Post reply on HN