Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

131–140 of 245 posts

Re: Facebook's git repo is 54GB

#131

Meh. I'm working on a comparably small project (~40 developers), and we're over 16GB. Mostly because we want a 100% reproducible build environment, so a complete build environment (compilers + IDE + build system) is all checked into the repro.

IDE checked into the repo eh? For some reason I kinda like that idea. So portable... if it works.

Re: Facebook's git repo is 54GB

#132
post #63

Earlier quoted context omitted.

Yes, and an intern would be subject to a code review before pushing to master - no way would they have write access to the "master" repo.

Yes, but the new intern would be able to read all the source and "secret sauces". I doubt that an intern on Google would've access to the search codebase. I'd wager that only a handful of trusted employees have access to that codebase.

If you don't trust your employees why did you hire them? Besides, the source code isn't where FB's money is at, that would be their users.

Re: Facebook's git repo is 54GB

#133
gosh, last time I had trouble with 8GB data checking in, it's very memory hungry when the data set is big and then you need check them in all at once, how much memory on the server side you need when you want to 'git add .' all the repo of 54GB?

what about a re-index or something, will that take forever?

I worry at such size the speed will suffer, I feel git is comfortable with probably a few GBs only?

anyway it's good to know that 54GB still is usable!

Re: Facebook's git repo is 54GB

#134
post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

git can not really deal with binary files well, if you have lots of png/gif/mov/mp3/jpg/etc to track git is not the option for you I think. git excels at source code, in txt format that is.

Re: Facebook's git repo is 54GB

#135
post #69
post #41

Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

I wonder, if this is the way a majority of big businesses do things, how come we don't see more leaks of entire codebases? It'd be trivial to put something up on TPB and just share all the code, but I don't see things like that happening. I also doubt that every single employee with access to the code has the moral standards not to do this. There must be something else keeping them from doing it.

Re: Facebook's git repo is 54GB

#136
post #82
post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

Facebook now has 7000 employees and is 10 years old. Each employee would have had to write 14 kiB of code (357 LOC with 40 (random guess of mine) characters per LOC) every day during this 10 years to produce 8 GiB of code. (Obviously assuming the repository contains only non-compressed code and only one version of everything and no metadata and...)

If they've got memcached with their own patches, linux with their own patches, Hadoop with their own patches, etc. and tons of translations I can see 8 gigs of text.

Re: Facebook's git repo is 54GB

#137
post #107
post #87

Earlier quoted context omitted.

>Everyone having access to everything must be worth the security trade-off. I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Having only the code wouldn't be enough. You need to also replicate comparable maintenance staff & data center capability. Even then you'll just have a clone of the tech, but nobody will actually use your service. There's so much more that makes facebook what it is. I think they may even be able to open source their tech with a net positive effect.

I think a lot of their value comes in the data the collect about users and how they use it to sell ads to vendors. That source code would be worth a lot. I'm not speaking about replicating a Facebook on your own servers, I agree there it would be futile. But how they run their business can probably be derived from their source code.

Re: Facebook's git repo is 54GB

#138
post #109
post #87

Earlier quoted context omitted.

>Everyone having access to everything must be worth the security trade-off. I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Lets say you managed to sneak out the code from facebook. You take the logo, draw a red cross over it and scribble "ProAmbook" below. You push it live. Now what? How do you get users? "We are just like facebook - only your friends aren't here" probably wouldn't get users excited. And if you somehow DID manage to get users, don't you think there are "watermarks" in the code, that they could detect and sue you to death…

I think a lot of their value comes in the data the collect about users and how they use it to sell ads to vendors. That source code would be worth a lot. I'm not speaking about replicating a Facebook on your own servers, I agree there it would be futile. But how they run their business can probably be derived from their source code. I do like the idea of ProAmBook though... :)

Re: Facebook's git repo is 54GB

#140
post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

Absolutely! BF4 with the expansion packs is > 40GB as a compiled binary and compressed assets. The source (with no history) must be gigantic.
Post reply on HN