Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

Post reply on HN
Print view View original

191–200 of 245 posts

Re: Facebook's git repo is 54GB

#191

Post by jonknee »

ProAm wrote: post #87
rl3 wrote: post #69

Earlier quoted context omitted.

It should be possible to restrict each employee's access to specific parts of the repository. However, I can't really see Facebook doing that. Everyone having access to everything must be worth the security trade-off. On the other hand, I suppose it's debatable whether it would be a trade-off at all.

>Everyone having access to everything must be worth the security trade-off. I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Facebook has never been defined by its code. They have had source leak before and no one has cared--it's PHP (heavily customized at that) and not of much worth outside of casual curiosity. The trick is to get a billion plus people to give you the details of your life every day.

Re: Facebook's git repo is 54GB

#193

Post by eridius »

kahoon wrote: post #91
rl3 wrote: post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

Git does have support for "shallow" clones, that don't pull down the full history but only the recent history.

Re: Facebook's git repo is 54GB

#194

Post by jzwinck »

ulisesrmzroche wrote: post #188
indygreg2 wrote: post #99

Earlier quoted context omitted.

SOA isn't a magic bullet. What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos). Multiple copies leads to copies getting out of sync. You (likely) lose the ability to perform a single atomic c…

You're coupling your 3rd party dependencies too tightly with your app logic, so that's why its so brittle. Start wrapping those functions.

When you are done writing boilerplate to wrap all the third-party functions, don't forget to write equivalent documentation too. And tests. Or are you just going to use the same function names? If the latter it's a pointless exercise, because it's unlikely you can shoehorn A different vendor library into the exact same API later.

Re: Facebook's git repo is 54GB

#195

Post by greggman »

eridius wrote: post #193
kahoon wrote: post #91

Earlier quoted context omitted.

But they surely don't use git for that, right? In scenarios like this a versioning system that does not track all history locally would be a better fit.

Git does have support for "shallow" clones, that don't pull down the full history but only the recent history.

shallow clones the only issue for games. Another problem is, unlike code, art assets usually can't be merged. You can't merge 2 photoshop texture files or 2 edits to a 3d character made in Maya.

So, you need some central system to manage the assets so that people know "hey, don't edit this file right now because so-and-so is editing it".

Ideally you'd like to know this BEFORE you start editing. In other words, you don't want to spent 15-60 minutes editing something and only on saving or trying to check in get told "hey, sorry but someone else was editing this file so you'll have to discard your changes and start over". Some editors are better at this than others.

You could try to write something on outside of git to help with this but why when P4 already provides this?

Re: Facebook's git repo is 54GB

#196

Post by patrickg »

samcasas wrote: post #10
_kushagra wrote: post #5

Earlier quoted context omitted.

Why's it bad to store source code on Git ?

Its not bad, is really nice, but Git has one problem, when you codebase is big, the process takes a long time, imagine git scanning those 8GB every time you do a commit, that is why Facebook was looking to port all their code to another VCS

I really can't support this claim. I have a repo of 111GB now and it works ok. Not slow at all (unless you do git gc or something like that).

Re: Facebook's git repo is 54GB

#197

Post by ProAm »

jonknee wrote: post #191
ProAm wrote: post #87

Earlier quoted context omitted.

>Everyone having access to everything must be worth the security trade-off. I would find this extremely hard to believe, especially at Facebook. At any software company, your code base is what defines you as a company; there is no way they'd let the good stuff sneak out like that.

Facebook has never been defined by its code. They have had source leak before and no one has cared--it's PHP (heavily customized at that) and not of much worth outside of casual curiosity. The trick is to get a billion plus people to give you the details of your life every day.

> The trick is to get a billion plus people to give you the details of your life every day.

How they capture this data and use it would also be in their source code, no? This is absolutely where Facebook gets its worth. I would assume this is what they would want to keep in a limited exposure set? I might be wrong, but this is why they hire the best engineers out there.

Re: Facebook's git repo is 54GB

#198

Post by eropple »

asdfaoeu wrote: post #185
yepguy wrote: post #177

Earlier quoted context omitted.

Strange as it sounds, git-annex doesn't really do file versioning very well.

Are you talking about git annex assistant or git annex? git annex does file versioning very nicely then again it doesn't work on Windows so that's probably not very useful for most game developers.

git-annex does file versioning, but it's extremely uncomfortable to use (and I say this as somebody totally comfortable with git) and I'd never expect an artist or other only-semi-technical person to use it even if it worked with Windows. Especially when Subversion or Perforce are right there.

Re: Facebook's git repo is 54GB

#199

Post by Myrmornis »

vmarsy wrote: post #110
d_j_b wrote: post #92

Earlier quoted context omitted.

that is a fascinating article, thank you. i like the idea of 'push karma'.

You're welcome! I really liked the 'push karma' too. I liked this idea too : "all engineers who contributed code must be available online during the push. The release system verifies this by contacting them automatically using a system of IRC bots; if an engineer is unavailable (at least for daily pushes), his or her commit will be reverted." That way, they are be able to react very quickly in case of a problem.

But this isn't a magic bullet...a commit can't always be reverted without conflicts, and even if reverted there's no guarantee that it leaves the codebase in a correct state. (Guess I should read the article you're talking about.)

Re: Facebook's git repo is 54GB

#200

Post by Too »

general_failure wrote: post #39

The worrying point here is the checkout of 8GB as opposed to the history size itself (46GB). If git is fast enough with SSD, this is hardly anything to worry about. I actually prefer monolithic repos (I realize that the slide posted might be in jest). I have seen projects struggle with submodules and splitting up modules into separate repos. People change something in their module. They don't test any upstream module…

As much as i hate clearcase i have to say it has some "interesting" features to deal with this. When you look at a repo you don't look at one unique version of the whole repo, you can create quite advanced "queries" such that for folder /src you look at your branch, for folder /3dpartylibs you look at version 5 and for folder /tests you look at head(latest). And since your working directory can be network attached (dynamic view) you dont have to "update" manually, head is like a living animal! Its like having subrepos built in.

While this is interesting it also requires a lot of discipline and almost one person dedicated as full time "dba" to not end up with spaghetti. Since there is no unique version number of the repo you have to store these queries and manually keep adding labels to be able to go back to an exact point of time.

It does have some uses like being able to run a new test on old code to find the first version when something broke or being able to change versions of external libraries or blob assets quickly but its hard to say if its worth it since it comes with so many other problems.

Post reply on HN