Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

pdfa.org

Post reply on HN
Print view View original

61–70 of 116 posts

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#61

Post by noname120 »

mort96 wrote: post #53
noname120 wrote: post #50

Earlier quoted context omitted.

Could you add compression and decompression speeds to your table?

I just did some interactive shell loops and globs to compress everything and output CSV which I processed into an ASCII table, so I don't exactly have a pipeline I can modify and re-run the tests with compression speeds added ... but I can run some more interactive shell-glob-and-loop-based analysis to give you decompression speeds: ~/tmp/pdfbench $ hyperfine --warmup 2 \ 'for x in zst/*; do zstd -d >/dev/null /dev/n…

Thanks a lot. Interestingly Brotli’s author mentioned here that zstd is 2× faster at decompressing, which roughly matches your numbers:

https://news.ycombinator.com/item?id=46035817

I’m also really surprised that gzip performs better here. Is there some kind of hardware acceleration or the like?

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#62

Post by mort96 »

terrelln wrote: post #59
mort96 wrote: post #41

Earlier quoted context omitted.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file size by more than a very small percent, like any compressor. Am I interpreting it correctly that you're doing something like `zstd -22 --ultra $FILE && wc -c $FILE.zst`? If you can reproduce this behavior, can you please file an issue with…

Okay now this is weird.

I can reproduce it just fine ... but only when compressing all PDFs simultaneously.

To utilize all cores, I ran:

    $ for x in *.pdf; do zstd "$x.zst" --ultra -22 & done; wait
(and similar for the other formats).

I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization:

    $ for x in *.pdf; do zstd "$x.zst" --ultra -22; done
That one file becomes 1.1M, and the total size of *.zst is 37M (competitive with Brotli, which is impressive given how much faster it is to decompress).

What's going on here? Surely '-22' disables any adaptive compression stuff based on system resource availability and just uses compression level 22?

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#63

Post by sgerenser »

mort96 wrote: post #48
order-matters wrote: post #46

Earlier quoted context omitted.

maybe that was too strongly worded but there was an expectation for zstd to outperform. So the fact it didnt means the result was unexpected. i generally find it helpful to understand why something performs better than expected.

Isn't zstd primarily designed to provide decent compression ratios at amazing speeds? The reason it's exciting is mainly that you can add compression to places where it didn't necessarily make sense before because it's almost free in terms of CPU and memory consumption. I don't think it has ever had a stated goal of beating compression ratio focused algorithms like brotli on compression ratio.

I actually thought zstd was supposed to be better than Brotli in most cases, but a bit of searching reveals you're right... Brotli, especially at the highest compression levels (10/11), often exceeds zstd at the highest compression levels (20-22). Both are very slow at those levels, although perfectly suitable for "compress once, decompress many" applications which the PDF spec is obviously one of them.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#64

Post by cortesoft »

HackerThemAll wrote: post #27
ksec wrote: post #8

Why not zstd?

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.

I can’t imagine the people actually doing the technical work don’t know about Zstandard.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#65

Post by kayodelycaon »

superkuh wrote: post #42

This is nice, but PDF jumped the shark already. It's no longer a document format that always looks the same everywhere. The inclusion of "Dynamic XFA (XML Form Architecture) PDF" in the spec made it so PDF is an unreliable format. The aformentioned is a PDF without content that pulls down all it's content from the web. It even still, ostensibly, supports Flash (swf) animations. In practice these "PDF"s are just empty…

Fortunately, XFA is deprecated. I haven’t seen one of those for a very long time.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#66

Post by dunham »

eru wrote: post #36
ndriscoll wrote: post #30

What is the point of using a generic compression algorithm in a file format? Does this actually get you much over turning on filesystem and transport compression, which can transparently swap the generic algorithm (e.g. my files are already all zstd compressed. HTTP can already negotiate brotli or zstd)? If it's not tuned to the application, it seems like it's better to leave it uncompressed and let the user decide w…

Well, if sanity had prevailed, we would have likely stuck to .ps.gz (or you favourite compression format), instead of ending up with PDF. Though we might still want to restrict the subset of PostScript that we allow. The full language might be a bit too general to take from untrusted third parties.

Don't you end up with PDF if you start with PS and restrict it to a subset? And maybe normalize the structure of the file a little. The structure is nice when you want to take the content and draw a bit more on the page. Or when subsetting/combining files.

I suspect PDF was fairly sane in the initial incarnation, and it's the extra garbage that they've added since then that is a source of pain.

I'm not a big fan of this additional change (nor any of the javascript/etc), but I would be fine with people leaving content streams uncompressed and running the whole file through brotli or something.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#67

Post by jsnell »

LoganDark wrote: post #44
eviks wrote: post #23

Earlier quoted context omitted.

Hey, they did all the work and more, trust them!!! > Experts in the PDF Association’s PDF TWG undertook theoretical and experimental analysis of these schemes, reviewing decompression speed, compression speed, compression ratio achieved, memory usage, code size, standardisation, IP, interoperability, prototyping, sample file creation, and other due diligence tasks.

I love when I perform all the due diligence tasks. You just can't counter that. Yes but, they did all the due diligence tasks. They considered all the factors. Every one. Think you have one they didn't consider? Nope.

But they didn't write "all". They wrote "other", which absolutely does not imply full coverage.

Maybe read things a bit more carefully before going all out on the snide comments?

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#68

Post by mort96 »

terrelln wrote: post #60
mort96 wrote: post #53

Earlier quoted context omitted.

I just did some interactive shell loops and globs to compress everything and output CSV which I processed into an ASCII table, so I don't exactly have a pipeline I can modify and re-run the tests with compression speeds added ... but I can run some more interactive shell-glob-and-loop-based analysis to give you decompression speeds: ~/tmp/pdfbench $ hyperfine --warmup 2 \ 'for x in zst/*; do zstd -d >/dev/null /dev/n…

Zstd should not be slower than gzip to decompress here. Given that it has inflated the files to be bigger than the uncompressed data, it has to do more work to decompress. This seems like a bug, or somehow measuring the wrong thing, and not the expected behavior.

It seems like zstd is somehow compressing really badly when many zstd processes are run in parallel, but works as expected when run sequentially: https://news.ycombinator.com/item?id=46723158

Regardless, this does not make a significant difference. I ran hyperfine again against a 37M folder of .pdf.zst files, and the results are virtually identical for zstd and gzip:

    +-------+-------+--------+-------+
    | gzip  | zstd  | brotli | xz    |
    +-------+-------+--------+-------+
    | 142ms | 165ms | 269ms  | 994ms |
    +-------+-------+--------+-------+
Raw hyperfine output:

    ~/tmp/pdfbench $ du -h zst2 gz xz br
     37M    zst2
     38M    gz
     38M    xz
     37M    br
    
    ~/tmp/pdfbench $ hyperfine ...
    Benchmark 1: for x in zst2/*; do zstd -d >/dev/null /dev/null /dev/null /dev/null

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#69

Post by mikkupikku »

dunham wrote: post #66
eru wrote: post #36

Earlier quoted context omitted.

Well, if sanity had prevailed, we would have likely stuck to .ps.gz (or you favourite compression format), instead of ending up with PDF. Though we might still want to restrict the subset of PostScript that we allow. The full language might be a bit too general to take from untrusted third parties.

Don't you end up with PDF if you start with PS and restrict it to a subset? And maybe normalize the structure of the file a little. The structure is nice when you want to take the content and draw a bit more on the page. Or when subsetting/combining files. I suspect PDF was fairly sane in the initial incarnation, and it's the extra garbage that they've added since then that is a source of pain. I'm not a big fan of t…

I thought PDFs can contain arbitrary PS.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#70

Post by Zekio »

mort96 wrote: post #62
terrelln wrote: post #59

Earlier quoted context omitted.

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file size by more than a very small percent, like any compressor. Am I interpreting it correctly that you're doing something like `zstd -22 --ultra $FILE && wc -c $FILE.zst`? If you can reproduce this behavior, can you please file an issue with…

Okay now this is weird. I can reproduce it just fine ... but only when compressing all PDFs simultaneously. To utilize all cores, I ran: $ for x in *.pdf; do zstd "$x.zst" --ultra -22 & done; wait (and similar for the other formats). I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization: $ for x in *.pdf; do zstd "$x.zst" --ultra -22; done That one fil…

doesn't zstd cap out at compression level 19?
Post reply on HN