Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running PostgreSQL on Compression-enabled ZFS

citusdata.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Running PostgreSQL on Compression-enabled ZFS

#41

Post by marshray »

jamhan wrote: post #40

Is it just me or is "Compression Ratio" a poor label for the graph in that article? Normally, when one uses "Compression Ratio", it is the opposite of those numbers, i.e. EXT3 storage would be 1:1, ZFS-LZJB would be 2:1 (not 0.5), and ZFS-gzip would be 3.33:1 (not 0.3). It's a small thing I know but it turns convention on its head in its current form. A better label would be perhaps "Storage Size Ratio".

I don't see a problem with them expressing the ratio as a decimal since it becomes a simple multiplier of the original file size 38GB x 0.3.

But it's downright misleading to show the vertical axis from something other than 0.0 to 1.0 when comparing ratios. They start it at 0.2. In reality, LZJB is saving 50% of the space whereas gzip saves 70%. But a naive glance at the graph implies gzip look roughly 3 times smaller/better than LZJB.

Classic "How to Lie with Statistics" stuff.* I would have expected better from an "analytics" database.

* Not saying they intend to lie here but it's representative of the classic text https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics

Re: Running PostgreSQL on Compression-enabled ZFS

#42

Post by jamhan »

marshray wrote: post #41
jamhan wrote: post #40

Is it just me or is "Compression Ratio" a poor label for the graph in that article? Normally, when one uses "Compression Ratio", it is the opposite of those numbers, i.e. EXT3 storage would be 1:1, ZFS-LZJB would be 2:1 (not 0.5), and ZFS-gzip would be 3.33:1 (not 0.3). It's a small thing I know but it turns convention on its head in its current form. A better label would be perhaps "Storage Size Ratio".

I don't see a problem with them expressing the ratio as a decimal since it becomes a simple multiplier of the original file size 38GB x 0.3. But it's downright misleading to show the vertical axis from something other than 0.0 to 1.0 when comparing ratios. They start it at 0.2. In reality, LZJB is saving 50% of the space whereas gzip saves 70%. But a naive glance at the graph implies gzip look roughly 3 times smaller…

If you read in any other article something like the following: "Taking Product X as having a baseline compression ratio of 1, Product Y had a compression ratio of 0.5 and Product Z had a compression ratio of 0.3", I'm pretty sure 99.9999% of the HN population would interpret that as Products Y and Z having worse compression than X, not better. That's my point.

Re: Running PostgreSQL on Compression-enabled ZFS

#43

Post by dpe82 »

richardkmichael wrote: post #24
cwsteinbach wrote: post #20

Earlier quoted context omitted.

I wasn't aware that Reiser4 supported compression. Thanks for pointing that out. As for why we chose to use ZFS instead of Btrfs, we feel that ZFS is closer to being in a state where an enterprise customer would be comfortable deploying it in production. This is due to the fact that ZFS has been in development for over a decade with many Solaris sites already using it in production, and Btrfs is still marked as "unst…

EDIT: I realize you said "near" and "closer" to production ready, but I think it's worth mentioning -- No FUD intended, but I don't consider ZFS on Linux production ready. Wanting to use ZFS, I recently started regularly reading their GitHub issues. There are deadlocks and un-importable pools in certain situations (hard-links being one: think rsync). I would not want production boxes in the same predicaments experien…

I'm using ZFS with L2ARC and write logs on an SSD on Ubuntu right now. Not sure I'd use it in production yet for the reasons you mention, but for things like my home workstation and office NAS it works great!

Re: Running PostgreSQL on Compression-enabled ZFS

#44

Post by marshray »

jamhan wrote: post #42
marshray wrote: post #41

Earlier quoted context omitted.

I don't see a problem with them expressing the ratio as a decimal since it becomes a simple multiplier of the original file size 38GB x 0.3. But it's downright misleading to show the vertical axis from something other than 0.0 to 1.0 when comparing ratios. They start it at 0.2. In reality, LZJB is saving 50% of the space whereas gzip saves 70%. But a naive glance at the graph implies gzip look roughly 3 times smaller…

If you read in any other article something like the following: "Taking Product X as having a baseline compression ratio of 1, Product Y had a compression ratio of 0.5 and Product Z had a compression ratio of 0.3", I'm pretty sure 99.9999% of the HN population would interpret that as Products Y and Z having worse compression than X, not better . That's my point.

This academic-looking paper (first hit I tried from Wikipedia) gives the standard definition of "compression ratio" as compressed/uncompressed size (section 4.2), consistent with the linked article.

I'm pretty sure you're impression of 99.9999% of the HN population is wrong.

Re: Running PostgreSQL on Compression-enabled ZFS

#45

Post by jamhan »

marshray wrote: post #44
jamhan wrote: post #42

Earlier quoted context omitted.

If you read in any other article something like the following: "Taking Product X as having a baseline compression ratio of 1, Product Y had a compression ratio of 0.5 and Product Z had a compression ratio of 0.3", I'm pretty sure 99.9999% of the HN population would interpret that as Products Y and Z having worse compression than X, not better . That's my point.

This academic-looking paper (first hit I tried from Wikipedia) gives the standard definition of "compression ratio" as compressed/uncompressed size (section 4.2), consistent with the linked article. I'm pretty sure you're impression of 99.9999% of the HN population is wrong.

Link?

OK, found this: http://en.wikipedia.org/wiki/Data_compression_ratio

Which includes this section on "Usage of the term": "There is some confusion about the term 'compression ratio', particularly outside academia and commerce. In particular, some authors use the term 'compression ratio' to mean 'space savings', even though the latter is not a ratio; and others use the term 'compression ratio' to mean its inverse, even though that equates higher compression ratio with lower compression."

So, my bad, however in my practical workplace experience the above (in italics) has been the case, hence the confusion.

Re: Running PostgreSQL on Compression-enabled ZFS

#46

Post by marshray »

skeletonjelly wrote: post #39
jaytaylor wrote: post #34

Earlier quoted context omitted.

FWIW, If you use one of the largest instance types (4x large or whatever), the VM will probably be on it's own host which would mean you're unlikely to have neighbors ;)

When benchmarking, it's best to remove assumptions based on "probably" though right?

It's cloud.

Re: Running PostgreSQL on Compression-enabled ZFS

#47

Post by marshray »

jamhan wrote: post #45
marshray wrote: post #44

Earlier quoted context omitted.

This academic-looking paper (first hit I tried from Wikipedia) gives the standard definition of "compression ratio" as compressed/uncompressed size (section 4.2), consistent with the linked article. I'm pretty sure you're impression of 99.9999% of the HN population is wrong.

Link? OK, found this: http://en.wikipedia.org/wiki/Data_compression_ratio Which includes this section on "Usage of the term": "There is some confusion about the term 'compression ratio', particularly outside academia and commerce. In particular, some authors use the term 'compression ratio' to mean 'space savings', even though the latter is not a ratio; and others use the term 'compression ratio' to mean its inverse…

Simple rule: If it's under 1.0 or expressed as a percentage under 100%, it's a compression ratio. If it's over 1.0, it's a compression factor.

Otherwise, it's not compressed. :-)

Re: Running PostgreSQL on Compression-enabled ZFS

#48

Post by reeses »

jaytaylor wrote: post #34
ars wrote: post #3

I wouldn't recommend doing benchmarking on a virtual server. You have no idea how busy the real server is, (noisy neighbors, etc), so it's impossible to have comparable results from benchmark to benchmark.

FWIW, If you use one of the largest instance types (4x large or whatever), the VM will probably be on it's own host which would mean you're unlikely to have neighbors ;)

You still have to deal with the storage fabric.

Re: Running PostgreSQL on Compression-enabled ZFS

#49

Post by cwsteinbach »

marshray wrote: post #41
jamhan wrote: post #40

Is it just me or is "Compression Ratio" a poor label for the graph in that article? Normally, when one uses "Compression Ratio", it is the opposite of those numbers, i.e. EXT3 storage would be 1:1, ZFS-LZJB would be 2:1 (not 0.5), and ZFS-gzip would be 3.33:1 (not 0.3). It's a small thing I know but it turns convention on its head in its current form. A better label would be perhaps "Storage Size Ratio".

I don't see a problem with them expressing the ratio as a decimal since it becomes a simple multiplier of the original file size 38GB x 0.3. But it's downright misleading to show the vertical axis from something other than 0.0 to 1.0 when comparing ratios. They start it at 0.2. In reality, LZJB is saving 50% of the space whereas gzip saves 70%. But a naive glance at the graph implies gzip look roughly 3 times smaller…

Author here. Believe it or not I originally had the compression ratio graph rotated 90 degrees, and had manually modified it to run from 0.00 to 1.00. Google docs for some god awful reason insists on starting at 0.2 by default. Anyway, when my colleagues reviewed a draft of this post they requested that I rotate the graph back, and in the process I forgot to reset the scale. Sorry for the confusion. It's fixed now. As for the definition of "compression ratio", I looked this up and went with the definition found here: http://en.wikipedia.org/wiki/Data_compression_ratio

I agree that it's kind of counterintuitive.

Re: Running PostgreSQL on Compression-enabled ZFS

#50

Post by BUGHUNTER »

petsos wrote: post #33

Can someone give us an overview of the state of ZFS on Linux? Last time I had checked it was implemented over fuse. Has this changed?

ZoL looks pretty good - unfortunately if you want Samba on ZoL, of course with snapshots and ACLs, you will have problems, as ACL mapping is not implemented, if I understood things well. That is a real pitty, ZFS is great, Samba is great, Linux is great and having these three things working smoothly together without having to spend weeks of research on how to get it running would help many admins to finally get away from commercial clown & bloat systems. However, the groundwork is done and if we are lucky in 2014 the Linux + Samba4 + ZFS dreamteam will be available as a stable replacement.
Post reply on HN