Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

Post reply on HN
Print view View original

81–90 of 245 posts

Re: Facebook's git repo is 54GB

#81

Post by GregorStocks »

Touche wrote: post #71
general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

How does monolithic repos solve that. Surely people who fix bugs in a library aren't testing the entirety of Facebook every time (how long would that even take? Assuming they've even set such a thing up.)

I used to work at Facebook. They have servers that automatically run a lot of their test cases on every commit.

Re: Facebook's git repo is 54GB

#82

Post by danbruc »

danbruc wrote: post #3
VikingCoder wrote: post #2

Pay attention to the footnote: *8 GB plus 46 GB .git directory

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

Facebook now has 7000 employees and is 10 years old. Each employee would have had to write 14 kiB of code (357 LOC with 40 (random guess of mine) characters per LOC) every day during this 10 years to produce 8 GiB of code. (Obviously assuming the repository contains only non-compressed code and only one version of everything and no metadata and...)

Re: Facebook's git repo is 54GB

#83

Post by rl3 »

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development.

Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc).

I wouldn't doubt there's some larger commercial game projects that have repository sizes exceeding 1TB.

Re: Facebook's git repo is 54GB

#84

Post by thathonkey »

general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

We use separate repos and it works out well. It's nice having separate Git histories that pertain to different areas of the codebase.

Our workflow covers all the potential problems you named (eg. scripts to keep everything up to date, tests that get run at build or push time after everything is already checked out from the individual repos, etc.).

We've been running this way for over a year with literally zero issues.

Re: Facebook's git repo is 54GB

#86

Post by ableal »

joshstrange wrote: post #46
antimatter wrote: post #7

Didn't they switch to Mercurial? https://code.facebook.com/posts/218678814984400/scaling-merc...

Somebody asked that on Twitter and the OP responded with: >> At least according to the presentation by a Facebook engineer that I just watched, they're still on git. [0] [0] https://twitter.com/feross/status/459335105853804544

You can check out, but you can never leave?

Re: Facebook's git repo is 54GB

#87

Post by ProAm »

rl3 wrote: post #69
slig wrote: post #41

Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

>Everyone having access to everything must be worth the security trade-off.

I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Re: Facebook's git repo is 54GB

#88

Post by vmarsy »

SnakeDoc wrote: post #54
justincormack wrote: post #32

Earlier quoted context omitted.

I think they did, have heard similar figures from them before, all deployed as one file. Macroservices...

unlikely. they probably have a very dirty repo with tons of binaries, images, blah blah blah. It's highly unlikely they actually wrote 8GB of code, and the 46GB .git directory will be littered with binary blob changes, etc. This is really just to "impress" two people: 1) People who love Facebook 2) People who don't know anything about version control and/or how to do proper version control (no binaries in the scm).

I cited this interesting paper from Facebook engineers(2013) in another comment :

https://news.ycombinator.com/item?id=7648802

In 2011 they had 10 millions LoC, up to 500 commits a day,

but if we asssume the plots keeps going up like this, now in 2014 it can be pretty big.

Their binary was 1.5GB when the paper was written.

Re: Facebook's git repo is 54GB

#89

Post by [deleted] »

rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

[deleted]

Re: Facebook's git repo is 54GB

#90

Post by sisk »

rwmj wrote: post #61
infogulch wrote: post #22

Earlier quoted context omitted.

The new depth option [1] on git clone as of 1.9.x should make the .git dir much more reasonable. > --depth : Create a shallow clone with a history truncated to the specified number of revisions. If the OS and applications that are creating and modifying files are all honest about the file dates, it should be possible to only scan dates instead of reading out every file. Or even use ionotify-like events to track what…

I've wanted this option for so long :-) Are there any downsides to it, apart from the obvious one of 'git log' not going all the way back in time? I mean, do any commands break unexpectedly?

The --depth flag has been around for a long time but, previously, came with a number of limitations (cannot push from, pull from, or clone from a shallow clone). As of 1.9.0, those limitations have been removed.

https://github.com/git/git/commit/82fba2b9d39163a0c9b7a3a2f3...

Post reply on HN