Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

Post reply on HN
Print view View original

111–120 of 245 posts

Re: Facebook's git repo is 54GB

#111

Post by vmarsy »

dnlserrano wrote: post #31

Someone recently told me that Facebook had a torrent file that went around the company that people could use to download the entire codebase using a BitTorrent client. Is there any truth in this? I mean, the same guy that told me this, also said that the codebase size was about 50 times less than the one reported in this slide, so it may all be pure speculation.

If you're interested in the deployment process at Facebook, look at the link of a Facebook engineers paper I submitted in my other comment in this thread : https://news.ycombinator.com/item?id=7648802

"The deployed executable size is around 1.5 Gbytes, including the Web server and compiled Facebook application. The code and data propagate to all servers via BitTorrent, which is configured to minimize global traffic by exploiting cluster and rack affinity. The time needed to propagate to all the servers is roughly 20 minutes."

Re: Facebook's git repo is 54GB

#112

Post by com2kid »

Meh. I'm working on a comparably small project (~40 developers), and we're over 16GB.

Mostly because we want a 100% reproducible build environment, so a complete build environment (compilers + IDE + build system) is all checked into the repro.

Re: Facebook's git repo is 54GB

#113

Post by rl3 »

kahoon wrote: post #91
rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

As the other replies say, Perforce is dominant in commercial game development.

However, Perforce does have Git integration now, allowing for either a centralized or distributed version control model. Considering the popularity of Git, I wouldn't doubt smaller Perforce-based game projects are going the DVCS route.

Also, hypothetically speaking, consider if you had a game project that would eventually grow to 1-2TB in repository size. If you spent $100 per developer to augment each of their workstations with a dedicated 3TB hard drive, you would have an awesome level of redundancy using DVCS (plus all the other advantages). I know it's no replacement for cold, off-site backups, but it would still be nice.

Re: Facebook's git repo is 54GB

#115

Post by nuser »

indygreg2 wrote: post #99
leccine wrote: post #74

Earlier quoted context omitted.

Hello there, have you heard of service oriented architecture? You must be joking to justify a single repository with "easier to change". Your problem is that the code base must be tightly coupled if splitting the services out to different repos is not possible and you need to contribute to multiple repositories to get something done. I would say, the biggest change in Amazon's architecture was moving over to the serv…

SOA isn't a magic bullet. What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos). Multiple copies leads to copies getting out of sync. You (likely) lose the ability to perform a single atomic c…

The advantages you ascribe to monolithic repos aren't due to monolithic repos, they're due to comprehensive tests.

The multiple copies of a shared library argument is straight-up nonsense, because in the multiple repo scenario, there would be (one or more) repos of shared libraries. There wouldn't be copying. Unless the devs were morons.

Builds in multiple repo environments are clearly identifiable, it's just by a combination of SHAs instead of a single SHA. In practice, this is a non-issue.

Version clashes happen in every scenario. Unified repos create a horrific dependency hell, because it's impossible to have one service using version X of a lib while another service uses a version Y. Instead, if you want to update from X to Y, the entire codebase needs to get the update, no matter whether it needs it or not. It's a boondoggle.

These decisions, as near as I can tell, are not made because they're good or bad, they're essentially arbitrary, related to what the first few engineers did. If it starts as a giant ball of code, it will always be a giant ball of code. If it starts out well-organized, it might remain well-organized.

Re: Facebook's git repo is 54GB

#116

Post by orenmazor »

lnanek2 wrote: post #59
gavinpc wrote: post #18

In terms of engineering tradeoffs, this reminds me of a recent talk by Alan Kay where he says that to build the software of the future, you have to pay extra to get the hardware of the future today. [1] Joel Spolsky called it "throwing money at the problem" when, five years ago he got SSD's for everybody at Fog Creek just to deal with a slow build. [2] I don't use Facebook, and I'm not suggesting that they're buildin…

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

when you're scaling any given variable, the stock solution is almost never good enough. you end up hitting all kinds of limits.

Re: Facebook's git repo is 54GB

#118

Post by spoiler »

maaaats wrote: post #21
danbruc wrote: post #3

Earlier quoted context omitted.

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

The big .git directory is probably binary revisions? Is there any good way around that in git?

I'm a bit confused. Whenever I've used git on my projects, I'd make sure the binaries were excluded, using .gitignore

Don't other people do that, too? What's the benefit of having binaries stored? I've never needed that; I've never worked on any huge projects, so I might be missing something crucial.

Re: Facebook's git repo is 54GB

#119

Post by danudey »

natrius wrote: post #19
indygreg2 wrote: post #17

Earlier quoted context omitted.

They aim for a completely linear history. They may even have a policy of not allowing merge commits. It is described in various places on the internet. I like https://secure.phabricator.com/book/phabflavor/article/recom... because it and its sister articles on code review and revision control are terrific reads.

Dear everyone: you should be using Phabricator. It is Facebook's collected wisdom about software development expressed in software. It has improved my life substantially. The code review is better than Github's, and their linear, squashed commit philosophy has worked out much better than the way I used to do things.

We're using Phabricator at my company, and I'm getting to the point of encouraging people to use it and start seeing the benefits. I work on infrastructure, so when people come to me with issues, I find out what problem they're having and then get them to submit a ticket to me. I've also started creating tickets for issues and assigning them to other people to get them to take a look at it, and people seem pretty receptive.

It hasn't become part of everyone's workflow yet, but it's pretty useful.

Re: Facebook's git repo is 54GB

#120

Post by Joeri »

Touche wrote: post #108
lnanek2 wrote: post #59

Earlier quoted context omitted.

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

They built their own build system because once you are dealing with top engineers NIH sets in quickly and you write your own everything.

They have their own in-house version of just about every dev tool. http://phabricator.org/
Post reply on HN