Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

pdfa.org

Post reply on HN
Print view View original

71–80 of 116 posts

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#71

Post by mort96 »

Zekio wrote: post #70
mort96 wrote: post #62

Earlier quoted context omitted.

Okay now this is weird. I can reproduce it just fine ... but only when compressing all PDFs simultaneously. To utilize all cores, I ran: $ for x in *.pdf; do zstd "$x.zst" --ultra -22 & done; wait (and similar for the other formats). I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization: $ for x in *.pdf; do zstd "$x.zst" --ultra -22; done That one fil…

doesn't zstd cap out at compression level 19?

From the man page:

    --ultra: unlocks high compression levels 20+ (maximum 22), using a lot more memory.
Regardless, this reproduces with random other files and with '-9' as the compression level. I made a mastodon post about it here: https://floss.social/@mort/115940378643840495

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#73

Post by deepsun »

xxs wrote: post #9
ericpauley wrote: post #7

Some real cognitive dissonance in this article… “The PDF Association operates under a strict principle—any new feature must work seamlessly with existing readers” followed by introducing compression as a breaking change in the same paragraph. All this for brotli… on a read-many format like pdf zstd’s decompression speed is a much better fit.

yup, zstd is better. Overall use zstd for pretty much anything that can benefit from a general purpose compression. It's a beyond excellent library, tool, and an algorithm (set of). Brotli w/o a custom dictionary is a weird choice to begin with.

Brotli compresses my files way better, but it's doing it way slower. Anyway, universal statement "zstd is better" is not valid.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#74

Post by Thoreandan »

atiedebee wrote: post #33
dist-epoch wrote: post #25

Earlier quoted context omitted.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompr…

Does your source .pdf material have FlateDecode'd chunks or did you fully uncompress it?

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#75

Post by terrelln »

mort96 wrote: post #62
terrelln wrote: post #59

Earlier quoted context omitted.

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M | Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file size by more than a very small percent, like any compressor. Am I interpreting it correctly that you're doing something like `zstd -22 --ultra $FILE && wc -c $FILE.zst`? If you can reproduce this behavior, can you please file an issue with…

Okay now this is weird. I can reproduce it just fine ... but only when compressing all PDFs simultaneously. To utilize all cores, I ran: $ for x in *.pdf; do zstd "$x.zst" --ultra -22 & done; wait (and similar for the other formats). I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization: $ for x in *.pdf; do zstd "$x.zst" --ultra -22; done That one fil…

Yeah, `--adaptive` will enable adaptive compression, but it isn't enabled by default, so shouldn't apply here. But even with `--adaptive`, after compressing each block of 128KB of data, zstd checks that the output size is So it is very central to zstd that it will never emit a block that is larger than 128KB+3B.

I will try to reproduce, but I suspect that there is something unrelated to zstd going on.

What version of zstd are you using?

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#76

Post by deepsun »

ericpauley wrote: post #7

Some real cognitive dissonance in this article… “The PDF Association operates under a strict principle—any new feature must work seamlessly with existing readers” followed by introducing compression as a breaking change in the same paragraph. All this for brotli… on a read-many format like pdf zstd’s decompression speed is a much better fit.

Well, except for speed, compression algorithms need to be compared in terms of compression, you know.

Here's discussion by brotli's and zstd's staff:

https://news.ycombinator.com/item?id=19678985

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#77

Post by terrelln »

mort96 wrote: post #68
terrelln wrote: post #60

Earlier quoted context omitted.

Zstd should not be slower than gzip to decompress here. Given that it has inflated the files to be bigger than the uncompressed data, it has to do more work to decompress. This seems like a bug, or somehow measuring the wrong thing, and not the expected behavior.

It seems like zstd is somehow compressing really badly when many zstd processes are run in parallel, but works as expected when run sequentially: https://news.ycombinator.com/item?id=46723158 Regardless, this does not make a significant difference. I ran hyperfine again against a 37M folder of .pdf.zst files, and the results are virtually identical for zstd and gzip: +-------+-------+--------+-------+ | gzip | zstd |…

Ah I understand. In this benchmark, Zstd's decompression time is 284 MB/s, and Gzip's is 330 MB/s. This benchmark is likely dominated by file IO for the faster decompressors.

On the incompressible files, I'd expect decompression of any algorithm to approach the speed of `memcpy()`. And would generally expect zstd's decompression speed to be faster. For example, on a x86 core running at 2GHz, Zstd is decompressing a file at 660 MB/s, and on my M1 at 1276 MB/s.

You could measure locally either using a specialized tool like lzbench [0], or for zstd by just running `zstd -b22 --ultra /path/to/file`, which will print the compression ratio, compression speed, and decompression speed.

[0] https://github.com/inikep/lzbench

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#78

Post by mort96 »

terrelln wrote: post #75
mort96 wrote: post #62

Earlier quoted context omitted.

Okay now this is weird. I can reproduce it just fine ... but only when compressing all PDFs simultaneously. To utilize all cores, I ran: $ for x in *.pdf; do zstd "$x.zst" --ultra -22 & done; wait (and similar for the other formats). I ran this again and it produced the same 2M file from the source 1.1M file. However when I run without paralellization: $ for x in *.pdf; do zstd "$x.zst" --ultra -22; done That one fil…

Yeah, `--adaptive` will enable adaptive compression, but it isn't enabled by default, so shouldn't apply here. But even with `--adaptive`, after compressing each block of 128KB of data, zstd checks that the output size is So it is very central to zstd that it will never emit a block that is larger than 128KB+3B. I will try to reproduce, but I suspect that there is something unrelated to zstd going on. What version of…

'zstd --version' reports: "** Zstandard CLI (64-bit) v1.5.7, by Yann Collet **". This is zstd installed through Homebrew on macOS 26 on an M1 Pro laptop. Also of interest, I was able to reproduce this with a random binary I had in /bin: https://floss.social/@mort/115940378643840495

I was completely unable to reproduce it on my Linux desktop though: https://floss.social/@mort/115940627269799738

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#79

Post by wizzwizz4 »

itsdesmond wrote: post #54
dist-epoch wrote: post #25

Earlier quoted context omitted.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

> Pareto I don’t think you’re using that correctly.

It's correct use of Pareto, short for Pareto frontier, if the claim being made is "for every needed compression ratio, zstd is faster; and for every needed time budget, zstd is faster". (Whether this claim is true is another matter.)

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#80

Post by gcr »

If we're making breaking changes to PDFs, I'd love if the committee added a modern image format like JPEG-XL. In my experience, most disk usage of PDFs comes from images, not streams.

I keep a bunch of comics in PDF but JPEG-XL is by far the best way to enjoy them in terms of disk space.

Post reply on HN