Company I did a contract for last year has 8MB of (Java) source code and a 52MB SVN repo and make £40 million a year out of it...
We're doing something wrong.
Facebook's git repo is 54GB
121–130 of 245 posts
Re: Facebook's git repo is 54GB
#122I bet most of that size is made up from the various dependencies Facebook probably has, though I'm still surprised it's that large. I expected the background worker things, like the facial recognition system for tagging people, and the video re-encoding libs, to be housed on separate repositories. I also wonder if that size includes a snapshot of a subset of Facebook's Graph, so that each developer has a "mini-facebo…
Reminds me of a recent project we had. The code we wrote came in at about 150k after an npm and bower install it was at 192mb.
out of curiosity, why were (and if so, why?) you committing node_modules and the bower destination dir.?
Re: Facebook's git repo is 54GB
#123Earlier quoted context omitted.
I bet they check in (or have checked in at some point) 3rd party libraries, jar files, generated files, etc. I battle this every day at $dayjob and we have a multi GB subversion repository with a separate one for the 3rd party binaries. Svn handles this a bit better than the DVCSes, so just checking out the HEAD is smaller than the full history you get in git/hg, and you can clean up some crud a bit. It just lives on…
You can clone with a single revision in git. Still ends up being around 2X the size I believe since you have the objects stored for the single revision as well as the working tree.
2X the size of what? The raw files? Assuming little shared data, sure. But svn will make two copies of every file too.
Re: Facebook's git repo is 54GB
#124Earlier quoted context omitted.
Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.
Having used ant as a build system for Android projects, I don't blame them. In my admittedly limited experience (Windows 7 x64, ant, Android SDK) ant is terribly slow to build projects with multiple source library dependencies and throwing hardware at the problem doesn't speed it up that much.
I don't see how this is an Ant-specific issue. Ant is just calling into javac with a classpath parameter. The actual execution time spent in Ant should be minimal.
Re: Facebook's git repo is 54GB
#125Earlier quoted context omitted.
Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.
> Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revi…
Is that all actually code? Each commit would have to add an average of 5KB compressed so maybe 20KB of brand new code.
Re: Facebook's git repo is 54GB
#126Earlier quoted context omitted.
The big .git directory is probably binary revisions? Is there any good way around that in git?
I'm a bit confused. Whenever I've used git on my projects, I'd make sure the binaries were excluded, using .gitignore Don't other people do that, too? What's the benefit of having binaries stored? I've never needed that; I've never worked on any huge projects, so I might be missing something crucial.
Well, it depends. Images are for instance binaries where a text diff makes little sense, so you have a copy of each version of the image ever used. And many projects use programs where the files are binaries. For instance, I've been on a project where Flash were used and the files checked in. Or PhotoShop PSD files, .ai files etc.
Re: Facebook's git repo is 54GB
#127Earlier quoted context omitted.
8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.
The big .git directory is probably binary revisions? Is there any good way around that in git?
When I can't avoid referring to binary blobs in git, I put them in a separate repo and link them with submodules. It keeps the main repo trim and the whole thing fast while still giving me end-to-end integrity guarantees.
I wrote https://github.com/polydawn/mdm/ to help with this. It goes a step further and puts each binary version in a separate root of history, which means you can pull down only what you need.
Re: Facebook's git repo is 54GB
#128Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…
I am working on a game where head is 1TB. On top of code and assets this size includes a few full builds and a full set of game-ready data (the data build process takes something ridiculous like 7 hours, so that's done on a server and it checks the result in). All in the same repository.
1TB is rather a lot. My previous record was 300GB and even that seemed a bit much. But it is very convenient having everything in one place, including all the stuff that you only need occasionally but is handy to have to hand, such as old builds of the game that you can just run in situ and installers for all the tools.
(I don't know what the entire repository size must be like, but many of the larger files have a limited history depth, so it's probably less than 5-10TB. So not exactly unimaginable, though I'm sure buying that much server-grade disk space - doubtless in duplicate - might cost more than I think.)
Re: Facebook's git repo is 54GB
#129So what? This probably means they're versioning data files they shouldn't be. I feel like this just exists here as a pissing contest.
Re: Facebook's git repo is 54GB
#130Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?
It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.
> It should be possible to restrict each employee's access to specific parts of the repository.
Not with git. The Hash chain mechanism requires the entire repo to generate valid chains, so it's all or nothing.