If I'm reading this right, with ZFS compression enabled I am seeing 1/3rd disk usage and 3x increase of speeds in query times just from switching the filesystem. Stats like that make me very skeptical. Does this mean that I can get a 3x increase in speed while cutting my disk space down by a third just by switching to ZFS? If so, why isn't everyone doing this?
The way I make sense of this is that you need fewer (slow) disk reads to get the same amount of data into RAM, so that might explain the speedup? I agree that it sounds too good to be true though.
Your read is correct. Once CPU time spent in decompression became less than disk wait time for the same data uncompressed, the reduced IO with compression started to win — sometimes massively. As powerful as processors are these days, results like these aren't impossible, or even terribly unlikely.
Consider the analogous (if simplified) case of logfile parsing, from my production syslog environment, with full query logging enabled:
# ls -lrt
...
-rw------- 1 root root 828096521 Apr 22 04:07 postgresql-query.log-20130421.gz
-rw------- 1 root root 8817070769 Apr 22 04:09 postgresql-query.log-20130422
# time zgrep -c duration postgresql-query.log-20130421.gz
19130676
real 0m43.818s
user 0m44.060s
sys 0m6.874s
# time grep -c duration postgresql-query.log-20130422
18634420
real 4m7.008s
user 0m9.826s
sys 0m3.843s