Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

Post reply on HN
Print view View original

211–220 of 245 posts

Re: Facebook's git repo is 54GB

#211

Post by masklinn »

Dylan16807 wrote: post #125
masklinn wrote: post #42

Earlier quoted context omitted.

> Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revi…

Is that all actually code? Each commit would have to add an average of 5KB compressed so maybe 20KB of brand new code.

> Is that all actually code?

Of course not, it's an actual project. There is code, there are data files (text and binary) and there are assets.

Re: Facebook's git repo is 54GB

#212

Post by pepijndevos »

rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

I wonder what the biggest code base in the world is.

Like you say, games include all sorts of binary assets. Any idea how much actual code is there?

The Linux kernel is only 175MB https://git.wiki.kernel.org/index.php/GitBenchmarks#Estimate...

The F22 has some 1.7 million LOC http://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor#cit...

This graph shows some pretty big things http://dailyinfographic.com/wp-content/uploads/2013/10/1276_...

Re: Facebook's git repo is 54GB

#213

Post by taeric »

JetSpiegel wrote: post #178
theelfismike wrote: post #150

Earlier quoted context omitted.

With Android, ant makes naive assumptions. For example, most open source library projects that you include in a project don't change from build to build, but ant dutifully recompiles them each time instead of caching the output until the files in that project are changed or I manually clean the build output.

That surely means Ant is make, only worse.

"Only worse" is probably not fair. Though, is it really surprising that a newer make file degenerates into the same problems as old ones?

Which will then lead down a path of a set of scripts/utilities on top of said system to standardize on a set of targets and deal with known issues. And suddenly we have reinvented another old tool, autotools. We'll probably find ourselves in their troubles soon enough.

Re: Facebook's git repo is 54GB

#214

Post by taeric »

Crito wrote: post #204
stuhood wrote: post #181

Earlier quoted context omitted.

That only applies to deployment. You're not building these services from the ground up: they're all going to have common libraries that need to stay up to date.

These are all solved problems. You create a package system that allows you to specify versioned dependencies to other packages. Your build and deployment systems can then build your package even though it depends on code that lives in other repositories owned by different teams. Hell, this even works across different versioning control systems; one team can be lagging along on SVN, another can have packages in P4, an…

As others have noted elsewhere, this "solution" has its own problems if you are rapidly moving. Which I don't think anyone can claim facebook hasn't been doing.

So, yes, if you are able to control growth enough that you can make this happen, it is attractive. If you can't, then this leads to a version of the diamond problem in project dependencies. And is not fun.

Re: Facebook's git repo is 54GB

#215

Post by taeric »

Oompa wrote: post #202
rcxdude wrote: post #33

Earlier quoted context omitted.

It becoms a lot harder to keep everything in sync, especially if internal interfaces change frequently. At facebook scale though it's probably a good idea to defined boundaries between areas in the application better.

You end up with less developers having to pull & merge/rebase if you have things in separate repos. Individual libraries/dependencies get worked on by themselves, with an API that other applications use. Then the other apps just bump a version number and get newer code.

The problem with this, is that you are assuming the APIs change in some sort of odd isolation to the parts that use them.

That is, the reason an API changes is because a use site has need of a change. So, at a minimum, you need to make that change and test it against that site in a somewhat atomic commit.

Then, if the change has any affect on other uses, you need a good way to test that change on them at the same time. Otherwise, they will resist pulling this change until it is fixed.

Add in more than a handful of such use sites, and suddenly things are just unmanageable in this "manageable" situation.

Not that this is "easy" in a central repo. But at least with the source dependency, you can get a compiler flag at every place an API change breaks something.

And, true, you can do this with multiple repos, too. But every attempt I have seen to do that just uses a frighteningly complicated tool to "recreate" what looks like a single source tree out of many separate ones. (jhbuild, and friends)

So, if there is a good tool for doing that, I'd certainly love to hear about it.

Re: Facebook's git repo is 54GB

#216

Post by JetSpiegel »

taeric wrote: post #213
JetSpiegel wrote: post #178

Earlier quoted context omitted.

That surely means Ant is make, only worse.

"Only worse" is probably not fair. Though, is it really surprising that a newer make file degenerates into the same problems as old ones? Which will then lead down a path of a set of scripts/utilities on top of said system to standardize on a set of targets and deal with known issues. And suddenly we have reinvented another old tool, autotools. We'll probably find ourselves in their troubles soon enough.

There was a link submitted here (I can't find it now) a few weeks ago that talked exactly about that. Most build systems are just reimplementations of make, which makes them worse , because make has been battle tested for ages.

Re: Facebook's git repo is 54GB

#217

Post by andrewjshults »

ianphughes wrote: post #28
natrius wrote: post #19

Earlier quoted context omitted.

Dear everyone: you should be using Phabricator. It is Facebook's collected wisdom about software development expressed in software. It has improved my life substantially. The code review is better than Github's, and their linear, squashed commit philosophy has worked out much better than the way I used to do things.

It looks pretty great. How does it compare to Atlassian products (if you have used any)?

Mixed bag. The code review part is much better than stash and significantly better than crucible. Namely, diff of diffs makes reviewing changes based on comments infinitely easier (especially on large reviews). We installed phabricator just for the code review piece initially. Repo browsing is about on par with stash, but it doesn't seem to experience the horrific slow downs that our stash server does. We don't use the tasks because a number of non engineering roles also use JIRA and the tasks functions in phabricator don't have nearly the depth of security and workflow options we need.

Re: Facebook's git repo is 54GB

#218

Post by andrewjshults »

ianphughes wrote: post #28
natrius wrote: post #19

Earlier quoted context omitted.

Dear everyone: you should be using Phabricator. It is Facebook's collected wisdom about software development expressed in software. It has improved my life substantially. The code review is better than Github's, and their linear, squashed commit philosophy has worked out much better than the way I used to do things.

It looks pretty great. How does it compare to Atlassian products (if you have used any)?

Mixed bag. The code review part is much better than stash and significantly better than crucible. Namely, diff of diffs makes reviewing changes based on comments infinitely easier (especially on large reviews). We installed phabricator just for the code review piece initially. Repo browsing is about on par with stash, but it doesn't seem to experience the horrific slow downs that our stash server does. We don't use the tasks because a number of non engineering roles also use JIRA and the tasks functions in phabricator don't have nearly the depth of security and workflow options we need.

Re: Facebook's git repo is 54GB

#219

Post by nawitus »

greggman wrote: post #195
eridius wrote: post #193

Earlier quoted context omitted.

Git does have support for "shallow" clones, that don't pull down the full history but only the recent history.

shallow clones the only issue for games. Another problem is, unlike code, art assets usually can't be merged. You can't merge 2 photoshop texture files or 2 edits to a 3d character made in Maya. So, you need some central system to manage the assets so that people know "hey, don't edit this file right now because so-and-so is editing it". Ideally you'd like to know this BEFORE you start editing. In other words, you do…

Even better would be collaborative real-time editing.

Re: Facebook's git repo is 54GB

#220

Post by skj »

slig wrote: post #63
TD-Linux wrote: post #50

Earlier quoted context omitted.

Yes, and an intern would be subject to a code review before pushing to master - no way would they have write access to the "master" repo.

Yes, but the new intern would be able to read all the source and "secret sauces". I doubt that an intern on Google would've access to the search codebase. I'd wager that only a handful of trusted employees have access to that codebase.

You are correct for the search codebase - it is separate from the main tree, as is a lot of the core ad placement code. If you aren't working on those projects, you don't have access.
Post reply on HN