Why Perforce is more scalable than Git
gandolf.homelinux.org
Why Perforce is more scalable than Git
1–10 of 48 posts
Re: Why Perforce is more scalable than Git
#2A an article titled "Why" ..... should usually explain WHY. This article, if anything, just asserts that git IS un-scalable. The closest thing to a "Why" is this "Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo."
Which may be true, but this article is pretty thin
Re: Why Perforce is more scalable than Git
#3A an article titled "Why" ..... should usually explain WHY. This article, if anything, just asserts that git IS un-scalable. The closest thing to a "Why" is this "Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo." Which may be true, but this article is pretty thin
Are you looking for "why" as in "what are the architecture problems" or as in "what are the scenarios and metrics demonstrating the case"?
I can give you the latter - git does not support partial checkouts. If your repository is 80Gb and you only need 5Gb to work with you will have to get all 80Gb over the network and it will be about 16 times as slow as it needs to be. The same problem does not exist in perforce - you can get any subset you wish.
Linus was pinned on this in his talk he gave at Google and his repsonse was "well, you can create multiple repostiroies and then write scripts on top to manage all this". To his credit he admitted the problem, but then he doesn't seem to care enough to do anything about it.
Re: Why Perforce is more scalable than Git
#4I actually go along with the "put any binary or tool you need to make this run in the repository" school of thought, but I don't stuff generated files in the repository.
I'm not sure "Git doesn't handle this dysfunctional source control usage" is really a valid complaint.
Re: Why Perforce is more scalable than Git
#5This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer fast, and traversing deep directory structures with lots of files becomes a scary problem..
I speak from experience; the game I'm working on at my studio has an in-house tool to optimally paletteize 2d assets for an embedded system. It's a computationally hard problem so a lot of intermediate data is generated to make the final build relatively fast.
This tool generates about five files, IIRC, and one folder, for each frame of animation. Multiply this by 30 frames per second for each sprite's anim. Multiply this by an entire game's assets. The result is 80,000+ files and folders checked into SVN. That's after the coder optimized the number of files needed slightly. It can take 20 minutes or more to do an SVN update because of all the checks needed.
The solution we really need is a versioning system for binary files. There aren't many of those. Maybe Perforce works. I don't know, I've never used it.
Re: Why Perforce is more scalable than Git
#6This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer…
Media assets are definitely an interesting problem. (Of course, then you're beyond "source control" and into something else, but that something else is important.)
Re: Why Perforce is more scalable than Git
#7I actually go along with the "put any binary or tool you need to make this run in the repository" school of thought, but I don't stuff generated files in the repository. I'm not sure "Git doesn't handle this dysfunctional source control usage" is really a valid complaint.
It's not really fair to call it "dysfunctional source control usage" (at least no fairer than the OA). The problem is with large asset files, which though almost always "generated" in the sense that people don't hand-craft them like code, are not necessarily a function of what is stored in the repo.
I'll agree with you that making git handle huge binary repositories speedily is probably not a worthwhile effort.
Re: Why Perforce is more scalable than Git
#8A an article titled "Why" ..... should usually explain WHY. This article, if anything, just asserts that git IS un-scalable. The closest thing to a "Why" is this "Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo." Which may be true, but this article is pretty thin
Are you looking for "why" as in "what are the architecture problems" or as in "what are the scenarios and metrics demonstrating the case"? I can give you the latter - git does not support partial checkouts. If your repository is 80Gb and you only need 5Gb to work with you will have to get all 80Gb over the network and it will be about 16 times as slow as it needs to be. The same problem does not exist in perforce - y…
You're right that Linus probably doesn't care about large binary files. I think that's just as well though. git can't be all things to all people. My gut feeling is that making git good for huge binary repositories would mean sacrificing 80% of what makes it so sweet for regular development.
I'd even go so far as to say that the needs of versioning large asset files and source code are so different that a system optimized for one will always be deficient for the other. Therefore I don't think the idea of "scripts on top" is so bad. Actually the ideal would be a set of porcelain commands built on two separate subsystems.
Re: Why Perforce is more scalable than Git
#9A an article titled "Why" ..... should usually explain WHY. This article, if anything, just asserts that git IS un-scalable. The closest thing to a "Why" is this "Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo." Which may be true, but this article is pretty thin
Are you looking for "why" as in "what are the architecture problems" or as in "what are the scenarios and metrics demonstrating the case"? I can give you the latter - git does not support partial checkouts. If your repository is 80Gb and you only need 5Gb to work with you will have to get all 80Gb over the network and it will be about 16 times as slow as it needs to be. The same problem does not exist in perforce - y…
Sure, but this problem applies to huge software products like Microsoft Office, not the sort of software that the average developer at the average company works on every day. So while Perforce may be good for that 1% of the population, Git probably meets the needs of everyone else better.
(BTW, there are repositories bigger than the Linux kernel that behave very well under Git. The Emacs git repository is an example, it has almost 30 years of history, weighs in at around 150M, but still performs fine. I know that I have never been paid to work on any application nearly this big, so I don't really worry about Git not meeting my needs.)
Re: Why Perforce is more scalable than Git
#10I have never worked on a 6GB repository, and apart from my (pretty solid) development background, I've spent the last 3 years doing code audits for huge software shops. The numbers in this post aren't compelling.
Consider also: is it possible that the multi-gigabyte repositories this guy's thinking of are byproducts of extremely crappy version control disciplines?