With binary files and git, am I saving all the previous versions in the tree locally (which could get huge fast), or just the diffs?
Binary diffs. But depending on the file, it might not really be of much help (I know for a fact that Adobe premiere projects for example are nearly completely different every time you save).
Folio uses git internally and as such is based on binary diffs. To tackle the issue of an ever-growing repository, we're looking into integrating git-lfs which supports offloading blobs from the repository.