Facebook's git repo is 54GB • Hacker News Board

Re: Facebook's git repo is 54GB

#141

Post by Peaker » Fri, Apr 25, 2014, 10:16 PM UTC

lnanek2 wrote: post #59

gavinpc wrote: post #18

In terms of engineering tradeoffs, this reminds me of a recent talk by Alan Kay where he says that to build the software of the future, you have to pay extra to get the hardware of the future today. [1] Joel Spolsky called it "throwing money at the problem" when, five years ago he got SSD's for everybody at Fog Creek just to deal with a slow build. [2] I don't use Facebook, and I'm not suggesting that they're buildin…

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

At my company, I wrote our own build system, because "make", "waf", "shake" and various others do not give any useful guarantee, and we've debugged cryptic under-specified dependencies way too many times. Make clean on a large repo and lack of automatic work sharing hurt too.

Also, auto detecting inputs rather than being forced to specify them is nice. Especially as virtually all input specs in Makefiles are wrong or incomplete.

Writing a build system is not such a big deal -- and outdoing the existing work is not very hard.

Re: Facebook's git repo is 54GB

#142

Post by cben » Fri, Apr 25, 2014, 10:16 PM UTC

Touche wrote: post #102

runT1ME wrote: post #94

Earlier quoted context omitted.

>People change something in their module. They don't test any upstream modules because it's not their problem anymore. If you use any sort of versioning this shouldn't ever cause a problem.

Yeah, doing it this way they can never make API incompatible changes without also fixing everything downstream... which effectively means once a library is popular enough it is locked at it's current API forever.

I've seen that happen at Google. At some point it's easier to write a new (differently named) library. Monolithic codebase + renames = poor man's versioning.

BUT this allows you to pay the prices of versioning (downstream burdened with rewriting => they never do => old code lives indefinitely) only in the worst cases. If done right (lots of tests, great CI infrastructure), fixing everything downstream is practical in many cases, and can be a win.

A subtler question is how this interplays with branching. You can't be expected to update downstream on every random/abandoned branch out there, only the head. Which deters people from branching because then it's their responsibility to keep up...

Re: Facebook's git repo is 54GB

#143

Post by pekk » Fri, Apr 25, 2014, 10:16 PM UTC

kyberias wrote: post #104

SnakeDoc wrote: post #62

Earlier quoted context omitted.

That sounds like either poor Sys Admin'ing and/or poor documentation... SCM should not have "clutter" in it, otherwise you wind up with an all-day download of 54GB of dead or useless garbage. The Kernel's repo is only a few GB's and it has MANY more changes and much more history than FB does...

You should realize that the kernel doesn't have image and other assets that FB might have. And the repo obviously is the right place for them.

"The" repo. As if they had no choice but to put everything for every aspect of the business into ONE giant repo.

Hopefully they actually have some separate sites, separate tools and separate libraries. Or could understand how to use submodules or something rather than literally putting everything in one huge repository.

Whether to put images and other assets into git repos is a separate decision.

Re: Facebook's git repo is 54GB

#144

Post by Peaker » Fri, Apr 25, 2014, 10:18 PM UTC

kahoon wrote: post #91

rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

It could use git-annex[1]?

[1] https://git-annex.branchable.com/

Re: Facebook's git repo is 54GB

#145

Post by pekk » Fri, Apr 25, 2014, 10:20 PM UTC

coherentpony wrote: post #129

So what? This probably means they're versioning data files they shouldn't be. I feel like this just exists here as a pissing contest.

It is supposed to justify the engineering effort they put into switching to Mercurial, then trying to make it "scale." (Rather than just using separate repositories to begin with, according to the design of the tool and best practices)

Re: Facebook's git repo is 54GB

#146

Post by namdnay » Fri, Apr 25, 2014, 10:23 PM UTC

maaaats wrote: post #21

danbruc wrote: post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

you're not going to merge binary files, so git isn't the right tool. the standard way is to use maven. git handles your sources, and anything binary (libs, resources etc) goes on the nexus (where it is versioned centrally) and is referenced in your poms: simple and powerful

Re: Facebook's git repo is 54GB

#147

Post by namdnay » Fri, Apr 25, 2014, 10:23 PM UTC

maaaats wrote: post #21

danbruc wrote: post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

you're not going to merge binary files, so git isn't the right tool. the standard way is to use maven. git handles your sources, and anything binary (libs, resources etc) goes on the nexus (where it is versioned centrally) and is referenced in your poms: simple and powerful

Re: Facebook's git repo is 54GB

#148

Post by namdnay » Fri, Apr 25, 2014, 10:24 PM UTC

maaaats wrote: post #21

danbruc wrote: post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

maven!

Re: Facebook's git repo is 54GB

#149

Post by [deleted] » Fri, Apr 25, 2014, 10:26 PM UTC

slig wrote: post #63

TD-Linux wrote: post #50

Earlier quoted context omitted.

Yes, and an intern would be subject to a code review before pushing to master - no way would they have write access to the "master" repo.

Yes, but the new intern would be able to read all the source and "secret sauces". I doubt that an intern on Google would've access to the search codebase. I'd wager that only a handful of trusted employees have access to that codebase.

[deleted]

Re: Facebook's git repo is 54GB

#150

Post by theelfismike » Fri, Apr 25, 2014, 10:28 PM UTC

diek wrote: post #124

georgemcbay wrote: post #72

Earlier quoted context omitted.

Having used ant as a build system for Android projects, I don't blame them. In my admittedly limited experience (Windows 7 x64, ant, Android SDK) ant is terribly slow to build projects with multiple source library dependencies and throwing hardware at the problem doesn't speed it up that much.

I don't see how this is an Ant-specific issue. Ant is just calling into javac with a classpath parameter. The actual execution time spent in Ant should be minimal.

With Android, ant makes naive assumptions.

For example, most open source library projects that you include in a project don't change from build to build, but ant dutifully recompiles them each time instead of caching the output until the files in that project are changed or I manually clean the build output.