Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

Post reply on HN
Print view View original

91–100 of 245 posts

Re: Facebook's git repo is 54GB

#91

Post by kahoon »

rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

Re: Facebook's git repo is 54GB

#92

Post by d_j_b »

vmarsy wrote: post #70
TD-Linux wrote: post #50

Earlier quoted context omitted.

Yes, and an intern would be subject to a code review before pushing to master - no way would they have write access to the "master" repo.

Indeed the process of facebook code deployment is pretty complex: https://www.facebook.com/publications/514128035341603/ And I assume they have extra requirements for interns to push code.

that is a fascinating article, thank you. i like the idea of 'push karma'.

Re: Facebook's git repo is 54GB

#93

Post by donaldguy »

SnakeDoc wrote: post #65
indygreg2 wrote: post #17

Earlier quoted context omitted.

They aim for a completely linear history. They may even have a policy of not allowing merge commits. It is described in various places on the internet. I like https://secure.phabricator.com/book/phabflavor/article/recom... because it and its sister articles on code review and revision control are terrific reads.

Linear codebase history? Why even use Git then... that's SVN stuff... we use Git now-a-days for a reason...

Last I knew, the FB mainline codebase was in fact still in SVN with git-svn dcommit (possibly hidden under arc) being how things land in "master" (and the revisions being merged quasi-manually to a release branch immediately before HH compile)

FB doesn't need to branch ... Gatekeeper (their A/B, feature flag system) really takes care of that concern logically

Re: Facebook's git repo is 54GB

#94

Post by runT1ME »

general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

>People change something in their module. They don't test any upstream modules because it's not their problem anymore.

If you use any sort of versioning this shouldn't ever cause a problem.

Re: Facebook's git repo is 54GB

#96

Post by SnakeDoc »

vmarsy wrote: post #88
SnakeDoc wrote: post #54

Earlier quoted context omitted.

unlikely. they probably have a very dirty repo with tons of binaries, images, blah blah blah. It's highly unlikely they actually wrote 8GB of code, and the 46GB .git directory will be littered with binary blob changes, etc. This is really just to "impress" two people: 1) People who love Facebook 2) People who don't know anything about version control and/or how to do proper version control (no binaries in the scm).

I cited this interesting paper from Facebook engineers(2013) in another comment : https://news.ycombinator.com/item?id=7648802 In 2011 they had 10 millions LoC, up to 500 commits a day, but if we asssume the plots keeps going up like this, now in 2014 it can be pretty big. Their binary was 1.5GB when the paper was written.

1.5GB binary that includes their dependencies (other binaries).

Re: Facebook's git repo is 54GB

#97

Post by stonemetal »

kahoon wrote: post #91
rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

Perforce tends to be big in game dev because it does better with repos full of giant blobs, has locking for working with them etc.

Re: Facebook's git repo is 54GB

#98

Post by eropple »

kahoon wrote: post #91
rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

Perforce is king in game development. It's also the only place I personally still use Subversion.

Re: Facebook's git repo is 54GB

#99

Post by indygreg2 »

leccine wrote: post #74
indygreg2 wrote: post #43

Earlier quoted context omitted.

Having all code in a single repository increases developer productivity by lowering the barrier to change. You can make a single atomic commit in one repository as opposed to N commits in M repositories. This is much, much easier than dealing with subrepos, repo sync, etc. Unified repos scales well up to a certain point before troubles arise. e.g. fully distributed VCS starts to break down when you have hundreds of M…

Hello there, have you heard of service oriented architecture? You must be joking to justify a single repository with "easier to change". Your problem is that the code base must be tightly coupled if splitting the services out to different repos is not possible and you need to contribute to multiple repositories to get something done. I would say, the biggest change in Amazon's architecture was moving over to the serv…

SOA isn't a magic bullet.

What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos).

Multiple copies leads to copies getting out of sync. You (likely) lose the ability to perform a single atomic commit. Furthermore, you've increased the barrier to change (and to move fast) by introducing uncertainty. Are Service X and Service Y using the latest/greatest version of the library? Why did my change to this library break Service Z? Oh, it's because Service Z lags 3 versions behind on this library and can't talk with my new version.

Unified repositories help eliminate the sync problem and make a whole class of problems that are detrimental to productivity and moving fast go away.

Facebook isn't alone in making this decision. I believe Google maintains a large Perforce repository for the same reasons.

Re: Facebook's git repo is 54GB

#100

Post by djur »

SnakeDoc wrote: post #65
indygreg2 wrote: post #17

Earlier quoted context omitted.

They aim for a completely linear history. They may even have a policy of not allowing merge commits. It is described in various places on the internet. I like https://secure.phabricator.com/book/phabflavor/article/recom... because it and its sister articles on code review and revision control are terrific reads.

Linear codebase history? Why even use Git then... that's SVN stuff... we use Git now-a-days for a reason...

There's a big difference between a linear history produced by actual linear development and one produced by `git rebase -i`. They both have the advantage of being easier to understand later, however.
Post reply on HN