Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Perforce is more scalable than Git

gandolf.homelinux.org

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Why Perforce is more scalable than Git

#21

Post by krschultz »

DenisM wrote: post #3
krschultz wrote: post #2

A an article titled "Why" ..... should usually explain WHY. This article, if anything, just asserts that git IS un-scalable. The closest thing to a "Why" is this "Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo." Which may be true, but this article is pretty thin

Are you looking for "why" as in "what are the architecture problems" or as in "what are the scenarios and metrics demonstrating the case"? I can give you the latter - git does not support partial checkouts. If your repository is 80Gb and you only need 5Gb to work with you will have to get all 80Gb over the network and it will be about 16 times as slow as it needs to be. The same problem does not exist in perforce - y…

The former. Nobody is twisting this guy to use git so I don't care if it doesn't work for his scenario, I care if it works for me. I've heard the "can't deal with large binary files/repos" many times before, I thought he was going to explain why that was the case.

Re: Why Perforce is more scalable than Git

#22

Post by jrockway »

DenisM wrote: post #19
silentbicycle wrote: post #18

Earlier quoted context omitted.

source control. Tracking large data dependencies requires very different techniques from tracking source code, config files, etc. which are predominantly textual. They're usually several orders of magnitude larger (so hashing everything is impractical), and merging things automatically is far more problematic. Doing data tracking really well involves framing it as a fundamentally different problem (algorithmically, i…

So, a medical student tells his professor that he does not want to study the ear problems but plans to specialize on the nose problems instead. "Really?" asked the prof, "so where exactly do you plan to specialize - the left nostril or the right nostril?". Similarly, git is speciailizing is small, text-only projects, but will not work well for a project involving some sort of grpaics-enriched GUI or large code base o…

Similarly, git is speciailizing is small, text-only projects, but will not work well for a project involving some sort of graphics-enriched GUI or large code base or both.

As I mention a few replies up, this covers about 99% of all programming projects. Very few projects have gigabytes of source code or graphics. If you just have a few hundred megabytes of graphics, git will do fine. If you only have 2 million lines of code, git will do fine.

Things larger than this are really rare, so it is not a problem that Git doesn't handle it well. Or rather, it's not a problem for very many people.

Re: Why Perforce is more scalable than Git

#23

Post by plinkplonk »

maybe someone who works in ane environment with large binary files that need to be tracked should just buckle down and write a tool appropriate for that scenario? (vs try to force fit git - a tool designed for a different set of usecases - )

Sounds like an opportunity for a startup or at least a cool open source project ("make something people want")

Re: Why Perforce is more scalable than Git

#24

Post by kmavm »

I should probably be somewhat careful what I say here, but I work for a commercial software company with one of the largest perforce databases (according to perforce) in existence. 3000 developers working on several million lines of code; I don't know how many MB offhand, but I'd guess O(1GB). Our central perforce server is a monster that we literally cannot keep fed with enough RAM, yet we frequently wait for a minute or two for a simple p4 submit, or p4 sync, to complete.

git on my local machine eats this tree for breakfast. I seceded from p4 a few months ago, and my workflow has never been smoother; I only ever interact with perforce for actual checkins, and to pull in other developers changes.

I guess the moral here is that to talk about whether something "scales", we need to be clear about what dimension we're trying to scale. In our experience, trying to get perforce to scale to huge numbers of developers has taken a lot of effort. Since git is numb to the number of developers, it has the potential to work better in our environment. YMMV.

Re: Why Perforce is more scalable than Git

#25

Post by DenisM »

jrockway wrote: post #22
DenisM wrote: post #19

Earlier quoted context omitted.

So, a medical student tells his professor that he does not want to study the ear problems but plans to specialize on the nose problems instead. "Really?" asked the prof, "so where exactly do you plan to specialize - the left nostril or the right nostril?". Similarly, git is speciailizing is small, text-only projects, but will not work well for a project involving some sort of grpaics-enriched GUI or large code base o…

Similarly, git is speciailizing is small, text-only projects, but will not work well for a project involving some sort of graphics-enriched GUI or large code base or both. As I mention a few replies up, this covers about 99% of all programming projects. Very few projects have gigabytes of source code or graphics. If you just have a few hundred megabytes of graphics, git will do fine. If you only have 2 million lines…

I recall we have started this discussion with a question of whether git scales or not. You position is essentially that git doesn't scale but you don't care.

In other words it seems to me that everyone here agrees about the facts - git does not scale. And then some people seem to care about it and some don't. Are we on the same page now?

Re: Why Perforce is more scalable than Git

#26

Post by michaelneale »

kmavm wrote: post #24

I should probably be somewhat careful what I say here, but I work for a commercial software company with one of the largest perforce databases (according to perforce) in existence. 3000 developers working on several million lines of code; I don't know how many MB offhand, but I'd guess O(1GB). Our central perforce server is a monster that we literally cannot keep fed with enough RAM, yet we frequently wait for a minu…

good to know - the local-git remote-other model seems to be pretty common (and one of the strengths of git - as its so easy to get started with git).

You also may as well say who it is ;)

Re: Why Perforce is more scalable than Git

#27

Post by jorgeortiz85 »

DenisM wrote: post #3
krschultz wrote: post #2

A an article titled "Why" ..... should usually explain WHY. This article, if anything, just asserts that git IS un-scalable. The closest thing to a "Why" is this "Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo." Which may be true, but this article is pretty thin

Are you looking for "why" as in "what are the architecture problems" or as in "what are the scenarios and metrics demonstrating the case"? I can give you the latter - git does not support partial checkouts. If your repository is 80Gb and you only need 5Gb to work with you will have to get all 80Gb over the network and it will be about 16 times as slow as it needs to be. The same problem does not exist in perforce - y…

How much code do you think Github stores? I'd wager at least several gigabytes.

Git -does- scale. It just doesn't scale as one huge repository. You do need tools on top to manage multiple projects. It'd be nice to have an open source Github.

Re: Why Perforce is more scalable than Git

#28

Post by shiro »

tptacek wrote: post #10

I have never worked on a 6GB repository, and apart from my (pretty solid) development background, I've spent the last 3 years doing code audits for huge software shops. The numbers in this post aren't compelling. Consider also: is it possible that the multi-gigabyte repositories this guy's thinking of are byproducts of extremely crappy version control disciplines?

To your question: I guess not. He just has a different development background. Productions like game dev or CG work require to track huge binary objects and source code in sync (I'd argue that, in these environments, any attempt to manage binary objects and sources in separate systems has eventually failed; here, source code includes shader programs, for example, which typically depend on the texture images they work on; they're inseparable. I've been there, done that)---but if you're in this industry and have managed to do it well, I'm eager to hear about your experience.

It seems that most of people discussing here agree that "Git is good for source version control, while Perforce is for asset version control". So I'm not really sure what people is arguing about.

Re: Why Perforce is more scalable than Git

#29

Post by shiro »

plinkplonk wrote: post #23

maybe someone who works in ane environment with large binary files that need to be tracked should just buckle down and write a tool appropriate for that scenario? (vs try to force fit git - a tool designed for a different set of usecases - ) Sounds like an opportunity for a startup or at least a cool open source project ("make something people want")

The main problem is that there are limited number of users that really need large asset tracking solutions, and these big players have spent many years and now have something more or less working.

To surpass the existing systems, you really need to work on the actual production assets, and in the environment where hundreds of artists/programmers are updating it daily. Such large "practical" production assets are not easily accessible to startups or open-source projects.

I don't say it is impossible, though; AlienBrain seems to manage sustaining business. And I've seen few people who are truly happy with their asset management systems.

Re: Why Perforce is more scalable than Git

#30

Post by TwoBit »

silentbicycle wrote: post #15
triplefox wrote: post #5

This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer…

On several occasions, Perforce has introduced merge errors when merging from one branch to another.* Without mentioning (or speaking for) my employer, we have a project with about ten branches (4-5 in current use), 50-60k changelists in the history, 2-3 gb of data. The project is 10+ years old, but I don't believe the full history has been kept. We use Perforce, but several major developers don't fully trust it, and…

You are telling Perforce to force-merge files without you having a say in it,and you are complaining that some of them didn't go right? Sorry, but you won't get better results from anything else if you have that policy.
Post reply on HN