Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

Post reply on HN
Print view View original

181–190 of 245 posts

Re: Facebook's git repo is 54GB

#181

Post by stuhood »

zhemao wrote: post #155
general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

Most big tech companies use a service-oriented architecture. The website is composed of many small services which communicate with each other over HTTP or RPC protocols. Each service has its own version control repo and is maintained by a different team. That's generally the best way of scaling up.

That only applies to deployment. You're not building these services from the ground up: they're all going to have common libraries that need to stay up to date.

Re: Facebook's git repo is 54GB

#182

Post by stuhood »

thathonkey wrote: post #84
general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

We use separate repos and it works out well. It's nice having separate Git histories that pertain to different areas of the codebase. Our workflow covers all the potential problems you named (eg. scripts to keep everything up to date, tests that get run at build or push time after everything is already checked out from the individual repos, etc.). We've been running this way for over a year with literally zero issues…

To get a log for a specific subdirectory, you just:

  git log -- my-teams-subdirectory

Re: Facebook's git repo is 54GB

#183

Post by blueblob »

Peaker wrote: post #141
lnanek2 wrote: post #59

Earlier quoted context omitted.

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

At my company, I wrote our own build system, because "make", "waf", "shake" and various others do not give any useful guarantee, and we've debugged cryptic under-specified dependencies way too many times. Make clean on a large repo and lack of automatic work sharing hurt too. Also, auto detecting inputs rather than being forced to specify them is nice. Especially as virtually all input specs in Makefiles are wrong or…

Perhaps that's true for your very specific use case but the same is likely to be true for other people using your build system. autodetection is great when it works and horrid when it fails.

Re: Facebook's git repo is 54GB

#184

Post by [deleted] »

Touche wrote: post #71
general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

How does monolithic repos solve that. Surely people who fix bugs in a library aren't testing the entirety of Facebook every time (how long would that even take? Assuming they've even set such a thing up.)

[deleted]

Re: Facebook's git repo is 54GB

#185

Post by asdfaoeu »

yepguy wrote: post #177
Peaker wrote: post #144

Earlier quoted context omitted.

It could use git-annex[1]? [1] https://git-annex.branchable.com/

Strange as it sounds, git-annex doesn't really do file versioning very well.

Are you talking about git annex assistant or git annex? git annex does file versioning very nicely then again it doesn't work on Windows so that's probably not very useful for most game developers.

Re: Facebook's git repo is 54GB

#186

Post by vhost- »

ycombasks wrote: post #135
rl3 wrote: post #69

Earlier quoted context omitted.

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

I wonder, if this is the way a majority of big businesses do things, how come we don't see more leaks of entire codebases? It'd be trivial to put something up on TPB and just share all the code, but I don't see things like that happening. I also doubt that every single employee with access to the code has the moral standards not to do this. There must be something else keeping them from doing it.

I'd wager they are more likely to get leaked to the black market for money where someone will keep it a secret in the hopes of finding some 0 days.

Re: Facebook's git repo is 54GB

#187

Post by ulisesrmzroche »

dman wrote: post #45
slig wrote: post #41

Am I missing something or this means a new intern working on a small feature, for instance, would have access to entire codebase?

That is a feature not a bug! Discoverability of code helps improve code quality and makes things less fragile.

Discoverability decreases as LOC increase, so that's not true.

Re: Facebook's git repo is 54GB

#188

Post by ulisesrmzroche »

indygreg2 wrote: post #99
leccine wrote: post #74

Earlier quoted context omitted.

Hello there, have you heard of service oriented architecture? You must be joking to justify a single repository with "easier to change". Your problem is that the code base must be tightly coupled if splitting the services out to different repos is not possible and you need to contribute to multiple repositories to get something done. I would say, the biggest change in Amazon's architecture was moving over to the serv…

SOA isn't a magic bullet. What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos). Multiple copies leads to copies getting out of sync. You (likely) lose the ability to perform a single atomic c…

You're coupling your 3rd party dependencies too tightly with your app logic, so that's why its so brittle. Start wrapping those functions.

Re: Facebook's git repo is 54GB

#189

Post by ulisesrmzroche »

rcxdude wrote: post #33
hk__2 wrote: post #23

Is there a reason why they keep everything in the same repo? Can’t you just split the code across multiple smaller repos?

It becoms a lot harder to keep everything in sync, especially if internal interfaces change frequently. At facebook scale though it's probably a good idea to defined boundaries between areas in the application better.

No, it doesn't. It's actually the complete opposite because you know, 'though shall separate those things that change frequently from those that don't'

Re: Facebook's git repo is 54GB

#190

Post by ulisesrmzroche »

djur wrote: post #100
SnakeDoc wrote: post #65

Earlier quoted context omitted.

Linear codebase history? Why even use Git then... that's SVN stuff... we use Git now-a-days for a reason...

There's a big difference between a linear history produced by actual linear development and one produced by `git rebase -i`. They both have the advantage of being easier to understand later, however.

That's not true dude. If you're following actual linear development you're likely to see a lot of 'poke build' and 'change css' and stuff like that. Git rebase -i gives you a change to rename your commit and organize it in a readable way.

So git rebase -i will be more readable, while actual linear history is always gibberish.

Post reply on HN