Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Perforce is more scalable than Git

gandolf.homelinux.org

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Why Perforce is more scalable than Git

#31

Post by ajb »

triplefox wrote: post #5

This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer…

There is an open source filesystem-based version control system: vesta (http://www.vestasys.org/). That's probably the sort of thing you need for this kind of work. It was developed by DEC and then Intel for chip development, and those guys check in binary blobs.

Vesta is a cool piece of technology in many ways. It has a pure functional build language, completely parallelisable builds with accurate caching between builds, etc. The guy who maintains it at Intel is a bit bitter because he can't understand why less capable version control or build systems are more popular. Which is really because vesta's advantages show up best in quite large projects, but once your project is that large, it's very hard to switch.

The web page looks moribund, but in fact it's still actively developed, and the developers hang out on IRC.

Re: Why Perforce is more scalable than Git

#33

Post by cousin_it »

jerf wrote: post #20
triplefox wrote: post #5

This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer…

Some of your repliers are beating around the bush a little, I'm going to come right out and say it: Source control systems are for controlling source code . They are built from top to bottom around the idea that they are storing text files. They are build around the idea that a text-based patch is a meaningful thing to use. They are build around the idea that there is a reasonable merge algorithm to use to merge two…

It can't be fixed without making them no longer great source control systems.

Why? Is there some deep architectural reason why Git can't perform like Perforce on large binary files? Something so deep it cannot ever be fixed? I've read through this whole thread and see no such reason yet, only hints that it exists.

Re: Why Perforce is more scalable than Git

#34

Post by balajis »

Best way I've found to do joint versioning of code with large datasets (whether binary or tab-delimited text):

1. Check in symbolic links to git. You can include the SHA-1 or MD5 in the file name.

2. Have those symbolic links point to your large out-of-tree directory of binary files.

3. rsync the out-of-tree directory when you need to do work off the server

4. Have a git hook check to see whether those files are present on your machine when you pull, and to update the SHA-1s in the symbolic link filenames when you push

By using symbolic links, at least you have the dependencies encoded within git, even if the big files themselves aren't there.

Re: Why Perforce is more scalable than Git

#35

Post by silentbicycle »

DenisM wrote: post #25
jrockway wrote: post #22

Earlier quoted context omitted.

Similarly, git is speciailizing is small, text-only projects, but will not work well for a project involving some sort of graphics-enriched GUI or large code base or both. As I mention a few replies up, this covers about 99% of all programming projects. Very few projects have gigabytes of source code or graphics. If you just have a few hundred megabytes of graphics, git will do fine. If you only have 2 million lines…

I recall we have started this discussion with a question of whether git scales or not. You position is essentially that git doesn't scale but you don't care. In other words it seems to me that everyone here agrees about the facts - git does not scale. And then some people seem to care about it and some don't. Are we on the same page now?

I'm a different person from jrockway, but my position is while git alone does not scale when tracking with very large binaries, it's an easily solved problem, as it is intentionally designed to work with other tools better suited to such tasks. Git + rysnc solves that problem for me in full, and should scale fine.

Git doesn't include a source code editor, profiler, or web browser, either. The Unix way of design involves the creation of independent programs that intercommunicate via files and/or pipes. The individual programs themselves can be small and simple due to not trying to solve several different problems at once, and since they communicate via buffered pipes, you get Erlang-style message-passing parallelism for free at the OS level.

Like I said, if you track the binary metadata (checksums and paths to fetch them from) in git but not the files themselves, your scaling problem goes away completely. If you have a personal problem with using more than one program jointly, that has nothing to do with git.

Re: Why Perforce is more scalable than Git

#36

Post by silentbicycle »

cousin_it wrote: post #33
jerf wrote: post #20

Earlier quoted context omitted.

Some of your repliers are beating around the bush a little, I'm going to come right out and say it: Source control systems are for controlling source code . They are built from top to bottom around the idea that they are storing text files. They are build around the idea that a text-based patch is a meaningful thing to use. They are build around the idea that there is a reasonable merge algorithm to use to merge two…

It can't be fixed without making them no longer great source control systems. Why? Is there some deep architectural reason why Git can't perform like Perforce on large binary files? Something so deep it cannot ever be fixed? I've read through this whole thread and see no such reason yet, only hints that it exists.

To my understanding: When git looks for changes it scans all the tracked files (checking timestamps before hashing) and hashing those that look changed. Commits generate patches for all changed files, then generate hashes for each file, then hash the set of files in each directory for a hash of the directory, repeated until the state of the entire repository is collected into one hash. This is normally pretty fast, and has a lot of advantages as a way to represent state changes in the project, but it also means that if the project has several huge binary files sitting about (or thousands of large binaries, etc.), it will have to hash them as well. This requires a full pass through the file any time that they look like they might have changed, new files are added, etc. (Mercurial works very similarly, though the internal data structures are different.) Running sha1 on a 342M file just took about 9 seconds on my computer; this goes up linearly with file size.

Git deals with the state of the tree as a whole, while Perforce, Subversion, and some others work at a file-by-file level, so they only need to scan the huge files when adding them, doing comparisons for updates, etc. (Updating or scanning for changes on perforce or subversion does scan the whole tree, though, which can be very slow.)

You can make git ignore the binary files via .gitignore, of course, and then they won't slow source tracking down anymore. You need to use something else to keep them in sync, though. (Rysnc works well for me. Unison is supposed to be good for this, as well.) You can still track the file metadata, such as a checksum and a path to fetch it from automatically, in a text file in git. It won't be able to do merges on the binaries, but how often do you merge binaries?

Re: Why Perforce is more scalable than Git

#37

Post by silentbicycle »

jerf wrote: post #20
triplefox wrote: post #5

This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer…

Some of your repliers are beating around the bush a little, I'm going to come right out and say it: Source control systems are for controlling source code . They are built from top to bottom around the idea that they are storing text files. They are build around the idea that a text-based patch is a meaningful thing to use. They are build around the idea that there is a reasonable merge algorithm to use to merge two…

Git IS a key/value store database. (A filesystem is a kind of database, too, of course.) There's a good summary of the internal data structures here -- http://www.kernel.org/pub/software/scm/git/docs/user-manual....

As I've said elsewhere in this thread, tracking metadata (path and sha1 hash) for large binary files in git and otherwise ignoring them via .gitignore works quite well. I'm pretty sure the "right tool" is either rysnc or something similar.

Re: Why Perforce is more scalable than Git

#38

Post by silentbicycle »

andr wrote: post #32

The article never actually says why Perforce is more scalable, except that you have "an IT department taking care of it."

Perforce only scans through everything when you update or scan for changed files, but mostly tracks changes file by file. Git (and mercurial) work with the state of the tree as a whole for most operations, because this makes handling searching, branching, merging, etc. much nicer, but it means that when you have gigantic binary files just sitting around, it takes time scanning them as well.

Re: Why Perforce is more scalable than Git

#39

Post by mst »

silentbicycle wrote: post #36
cousin_it wrote: post #33

Earlier quoted context omitted.

It can't be fixed without making them no longer great source control systems. Why? Is there some deep architectural reason why Git can't perform like Perforce on large binary files? Something so deep it cannot ever be fixed? I've read through this whole thread and see no such reason yet, only hints that it exists.

To my understanding: When git looks for changes it scans all the tracked files (checking timestamps before hashing) and hashing those that look changed. Commits generate patches for all changed files, then generate hashes for each file, then hash the set of files in each directory for a hash of the directory, repeated until the state of the entire repository is collected into one hash. This is normally pretty fast, a…

Well, rsync has a mode where you say "look, if the file size and timestamp are the same, please just assume it hasn't changed - I'm happy with this and am willing to accept that if that isn't sufficient any resulting problems are mine".

I wonder if having some way to tell git to do the same for files with a particular extension/in a particular directory would get us a decent amount of the way there?

Re: Why Perforce is more scalable than Git

#40

Post by silentbicycle »

mst wrote: post #39
silentbicycle wrote: post #36

Earlier quoted context omitted.

To my understanding: When git looks for changes it scans all the tracked files (checking timestamps before hashing) and hashing those that look changed. Commits generate patches for all changed files, then generate hashes for each file, then hash the set of files in each directory for a hash of the directory, repeated until the state of the entire repository is collected into one hash. This is normally pretty fast, a…

Well, rsync has a mode where you say "look, if the file size and timestamp are the same, please just assume it hasn't changed - I'm happy with this and am willing to accept that if that isn't sufficient any resulting problems are mine". I wonder if having some way to tell git to do the same for files with a particular extension/in a particular directory would get us a decent amount of the way there?

While I haven't checked the source, I'm pretty sure by default git doesn't re-hash any files unless the timestamps have changed. (I know Mercurial doesn't.)
Post reply on HN