Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

201–210 of 245 posts

Re: Facebook's git repo is 54GB

#201
post #23

Is there a reason why they keep everything in the same repo? Can’t you just split the code across multiple smaller repos?

When you split code across smaller repos, you gain a different problem: version management and potentially dependencies using different versions of the same project.

> When you split code across smaller repos, you gain a different problem: version management and potentially dependencies using different versions of the same project.

True, you do. What you gain is the ability of small pieces to move individually through API changes.

If your entire codebase is in one repo (as appears to be the case here), and you want to change an API, you must either do so in a backwards compatible way, and slowly eradicate any old callers, or change them all in one fell swoop.

By splitting to multiple repos, you can version them independently. Thus, a project can (hopefully temporarily) depend on the old API, which only gets bugfixes, while another project can depend on the new version.

The tricky bit is when you have one "binary" or something equivalent referring to two versions of a dependency. (Usually indirectly, i.e., A depends on B which depends on D v1, and A depends on C which depends on D v2, and D v1 and D v2 are incompatible.) You can't really do much about this, but if you keep your components small enough (think services with well separated interfaces) you should be able to keep the dependencies small enough as well.

Re: Facebook's git repo is 54GB

#202
post #33
post #23

Is there a reason why they keep everything in the same repo? Can’t you just split the code across multiple smaller repos?

It becoms a lot harder to keep everything in sync, especially if internal interfaces change frequently. At facebook scale though it's probably a good idea to defined boundaries between areas in the application better.

You end up with less developers having to pull & merge/rebase if you have things in separate repos.

Individual libraries/dependencies get worked on by themselves, with an API that other applications use. Then the other apps just bump a version number and get newer code.

Re: Facebook's git repo is 54GB

#203
post #128
post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

I am working on a game where head is 1TB. On top of code and assets this size includes a few full builds and a full set of game-ready data (the data build process takes something ridiculous like 7 hours, so that's done on a server and it checks the result in). All in the same repository. 1TB is rather a lot. My previous record was 300GB and even that seemed a bit much. But it is very convenient having everything in o…

WOW. BTW what game is that you are developing ?

Re: Facebook's git repo is 54GB

#204
post #155

Earlier quoted context omitted.

Most big tech companies use a service-oriented architecture. The website is composed of many small services which communicate with each other over HTTP or RPC protocols. Each service has its own version control repo and is maintained by a different team. That's generally the best way of scaling up.

That only applies to deployment. You're not building these services from the ground up: they're all going to have common libraries that need to stay up to date.

These are all solved problems. You create a package system that allows you to specify versioned dependencies to other packages. Your build and deployment systems can then build your package even though it depends on code that lives in other repositories owned by different teams. Hell, this even works across different versioning control systems; one team can be lagging along on SVN, another can have packages in P4, and yet another can have theirs in git, but they can all build against each others code.

It works absolutely brilliantly. Division of labor and responsibility becomes clear, repos stay manageable, large scale rewrites can happen safely, in piecemeal, over time... it really is the best way to do it.

Re: Facebook's git repo is 54GB

#205
post #74

Earlier quoted context omitted.

Hello there, have you heard of service oriented architecture? You must be joking to justify a single repository with "easier to change". Your problem is that the code base must be tightly coupled if splitting the services out to different repos is not possible and you need to contribute to multiple repositories to get something done. I would say, the biggest change in Amazon's architecture was moving over to the serv…

SOA isn't a magic bullet. What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos). Multiple copies leads to copies getting out of sync. You (likely) lose the ability to perform a single atomic c…

> "What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos)."

No, you have a notion of packages in your build system and deployment system.

You want to use FooWizz framework for your new service BarQuxer? Include FooWiz+=2.0 as a dependency of your service. The build system will then get the suitable package FooWiz when building your BarQuxer. Another team on the other side of the company also wants to use FooWiz? They do the exact same thing. There is never a need for FooWiz to be duplicated, anybody can build with that package as a dependency.

Re: Facebook's git repo is 54GB

#206
post #60
post #52

Earlier quoted context omitted.

Oh yeah, it's great when one person can break the build and stop all active development. It scales so well. Oh I know, to prevent any issues, let's protect ourselves with feature toggles. Oh and let's build a set of database driven rules to manage those toggles. Oh what about dependencies? Let's build a manager to manage the DAG of feature toggle dependencies. Need I go on? :-) You've replaced a relatively simple sys…

My team uses Phabricator without any feature toggles in our code. You land code onto master when it's ready to ship. Until then, you have features developed and reviewed on separate branches. I don't get how that's more or less fragile than merges.

I was addressing the idea of committing directly to master, protecting your code with feature toggles so it doesn't break things. Maybe I misunderstood the OP.

I think feature toggles can be extremely useful, but still develop in a branch and merge after review/qa.

Re: Facebook's git repo is 54GB

#207
post #128

Earlier quoted context omitted.

I am working on a game where head is 1TB. On top of code and assets this size includes a few full builds and a full set of game-ready data (the data build process takes something ridiculous like 7 hours, so that's done on a server and it checks the result in). All in the same repository. 1TB is rather a lot. My previous record was 300GB and even that seemed a bit much. But it is very convenient having everything in o…

WOW. BTW what game is that you are developing ?

Half-Life 3 I assume.

Re: Facebook's git repo is 54GB

#208
post #141

Earlier quoted context omitted.

At my company, I wrote our own build system, because "make", "waf", "shake" and various others do not give any useful guarantee, and we've debugged cryptic under-specified dependencies way too many times. Make clean on a large repo and lack of automatic work sharing hurt too. Also, auto detecting inputs rather than being forced to specify them is nice. Especially as virtually all input specs in Makefiles are wrong or…

Perhaps that's true for your very specific use case but the same is likely to be true for other people using your build system. autodetection is great when it works and horrid when it fails.

I use file system hooks to autodetect dependencies, so it should always work as long as the file system is the input of the build and not other channels.

Explicit input specifications are virtually never correct. #include scanners for example, are generally wrong because they do not express the dependency on the inexistence of the headers in previous include paths.

Re: Facebook's git repo is 54GB

#209
post #170

Earlier quoted context omitted.

No, git has delta compression. It only saves the changes.

Only when git repack operations run.

which you can run at any time. Git also runs a minor gc every time you do a git push, which does do some compression.

Re: Facebook's git repo is 54GB

#210
post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

Absolutely! BF4 with the expansion packs is > 40GB as a compiled binary and compressed assets. The source (with no history) must be gigantic.

That is mostly data and assets though.
Post reply on HN