Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

101–110 of 245 posts

Re: Facebook's git repo is 54GB

#101
post #91
post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

I think most use Perforce.

Re: Facebook's git repo is 54GB

#102
post #94

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

>People change something in their module. They don't test any upstream modules because it's not their problem anymore. If you use any sort of versioning this shouldn't ever cause a problem.

Yeah, doing it this way they can never make API incompatible changes without also fixing everything downstream... which effectively means once a library is popular enough it is locked at it's current API forever.

Re: Facebook's git repo is 54GB

#103
post #63

Earlier quoted context omitted.

Yes, and an intern would be subject to a code review before pushing to master - no way would they have write access to the "master" repo.

Yes, but the new intern would be able to read all the source and "secret sauces". I doubt that an intern on Google would've access to the search codebase. I'd wager that only a handful of trusted employees have access to that codebase.

So it is like the Coca Cola recipe? Only two people allowed to know it and not allowed to get the same flight? Yet everyone else able to mix and serve the syrup?

Re: Facebook's git repo is 54GB

#104
post #58

Earlier quoted context omitted.

Honestly, I prefer check-in everything shops. It's way too often otherwise some different Java version, or IDE version, or Maven central being down screws something up, or you have to wait a long time for a Chef recipe or disk image to give you the reference version. Half my day today was dealing with someone updating Java on half our continuous build system slave computers and breaking everything because it didn't h…

That sounds like either poor Sys Admin'ing and/or poor documentation... SCM should not have "clutter" in it, otherwise you wind up with an all-day download of 54GB of dead or useless garbage. The Kernel's repo is only a few GB's and it has MANY more changes and much more history than FB does...

You should realize that the kernel doesn't have image and other assets that FB might have. And the repo obviously is the right place for them.

Re: Facebook's git repo is 54GB

#105
post #94

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

>People change something in their module. They don't test any upstream modules because it's not their problem anymore. If you use any sort of versioning this shouldn't ever cause a problem.

I had a problem so I decided to use versioning, now I have a combinatorially exploding number of problems.

Re: Facebook's git repo is 54GB

#106
FB had previous scaling problems with git which they discussed in 2012 http://comments.gmane.org/gmane.comp.version-control.git/189...

It appears they are now using Mercurial and working on scaling that (also noted by several others in this discussion): https://code.facebook.com/posts/218678814984400/scaling-merc...

Re: Facebook's git repo is 54GB

#107
post #87
post #69

Earlier quoted context omitted.

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

>Everyone having access to everything must be worth the security trade-off. I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Having only the code wouldn't be enough. You need to also replicate comparable maintenance staff & data center capability. Even then you'll just have a clone of the tech, but nobody will actually use your service. There's so much more that makes facebook what it is. I think they may even be able to open source their tech with a net positive effect.

Re: Facebook's git repo is 54GB

#108
post #59
post #18

In terms of engineering tradeoffs, this reminds me of a recent talk by Alan Kay where he says that to build the software of the future, you have to pay extra to get the hardware of the future today. [1] Joel Spolsky called it "throwing money at the problem" when, five years ago he got SSD's for everybody at Fog Creek just to deal with a slow build. [2] I don't use Facebook, and I'm not suggesting that they're buildin…

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

They built their own build system because once you are dealing with top engineers NIH sets in quickly and you write your own everything.

Re: Facebook's git repo is 54GB

#109
post #87
post #69

Earlier quoted context omitted.

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

>Everyone having access to everything must be worth the security trade-off. I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Lets say you managed to sneak out the code from facebook. You take the logo, draw a red cross over it and scribble "ProAmbook" below. You push it live.

Now what? How do you get users? "We are just like facebook - only your friends aren't here" probably wouldn't get users excited.

And if you somehow DID manage to get users, don't you think there are "watermarks" in the code, that they could detect and sue you to death with?

Re: Facebook's git repo is 54GB

#110
post #92
post #70

Earlier quoted context omitted.

Indeed the process of facebook code deployment is pretty complex: https://www.facebook.com/publications/514128035341603/ And I assume they have extra requirements for interns to push code.

that is a fascinating article, thank you. i like the idea of 'push karma'.

You're welcome! I really liked the 'push karma' too.

I liked this idea too : "all engineers who contributed code must be available online during the push. The release system verifies this by contacting them automatically using a system of IRC bots; if an engineer is unavailable (at least for daily pushes), his or her commit will be reverted."

That way, they are be able to react very quickly in case of a problem.

Post reply on HN