Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

pdfa.org

Post reply on HN
Print view View original

101–110 of 116 posts

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#101

Post by Dylan16807 »

dchest wrote: post #34
dist-epoch wrote: post #25

Earlier quoted context omitted.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

Not with small files.

If that's about using predefined dictionaries, zstd can use them too.

If brotli has a different advantage on small source files, you have my curiosity.

If you're talking about max compression, zstd likely loses out there, the answer seems to vary based on the tests I look at, but it seems to be better across a very wide range.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#102

Post by [deleted] »

gcr wrote: post #93
Bolwin wrote: post #82

Earlier quoted context omitted.

Odd you should say that, as that's exactly what they've been discussing

No it's not. This article is about proposing Brotli as another possible '/Filter' for stream objects, like content streams (page drawing commands). Images are streams too, but unless you mean compressing raw pixel bytes in Brotli, there's no mention of a JPEG-XL or WEBP filter.

[deleted]

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#103

Post by eru »

dunham wrote: post #66
eru wrote: post #36

Earlier quoted context omitted.

Well, if sanity had prevailed, we would have likely stuck to .ps.gz (or you favourite compression format), instead of ending up with PDF. Though we might still want to restrict the subset of PostScript that we allow. The full language might be a bit too general to take from untrusted third parties.

Don't you end up with PDF if you start with PS and restrict it to a subset? And maybe normalize the structure of the file a little. The structure is nice when you want to take the content and draw a bit more on the page. Or when subsetting/combining files. I suspect PDF was fairly sane in the initial incarnation, and it's the extra garbage that they've added since then that is a source of pain. I'm not a big fan of t…

> Don't you end up with PDF if you start with PS and restrict it to a subset?

PDF is also a binary format.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#104

Post by ksec »

mort96 wrote: post #51
noname120 wrote: post #47

Earlier quoted context omitted.

Why not use a more widespread compression algorithm (e.g. gzip) considering that Brotli barely performs better at all? Sounds like a pain for portability

I'm not sold on the idea of adding compression to PDF at all, I'm not convinced that the space savings are worth breaking compatibility with older readers. Especially when you consider that you can just compress it in transit with e.g HTTP's 'Content-Encoding' without any special PDF reader support. (You can even use 'Content-Encoding: br' for brotli!) If you do wanna change PDF backwards-incompatibly, I don't think…

>But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

I may dislike Google. But my support of JPEG XL and Zstd has nothing to do with competition tech being Google at all. I simply think JPEG XL and Zstd are better technology.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#105

Post by xxs »

jeffbee wrote: post #28
dist-epoch wrote: post #25

Earlier quoted context omitted.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

Are you sure? Admittedly I only have 1 PDF in my homedir, but no combination of flags to zstd gets it to match the size of brotli's output on that particular file. Even zstd --long --ultra -22.

on max compression (11 vs zstd's 22) of text brotli will be around 3-4% denser... and a lot slower. Decompression wise zstd is over 2x faster.

The pdfs you have are already compressed with deflate (zip).

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#106

Post by Someone »

wizzwizz4 wrote: post #81
Someone wrote: post #40

Earlier quoted context omitted.

- inside the file, the compressor can be varied according to the file content. For example, images can use jpeg, but that isn’t useful for compressing text - when jumping from page to page, you won’t have to decompress the entire file

> inside the file, the compressor can be varied according to the file content. For example, images can use jpeg, but that isn’t useful for compressing text Okay, so we make a compressed container format that can perform such shenanigans, for the same amount of back-compat issues as extending PDF in this way. > when jumping from page to page, you won’t have to decompress the entire file This is already a thing with an…

> Okay, so we make a compressed container format that can perform such shenanigans, for the same amount of back-compat issues as extending PDF in this way.

Far from the same amount:

- existing tools that split PDFs into pages will remain working

- if defensively programmed, existing PDF readers will be able to render PDFs containing JPEG XL images, except for the images themselves.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#107

Post by xxs »

deepsun wrote: post #73
xxs wrote: post #9

Earlier quoted context omitted.

yup, zstd is better. Overall use zstd for pretty much anything that can benefit from a general purpose compression. It's a beyond excellent library, tool, and an algorithm (set of). Brotli w/o a custom dictionary is a weird choice to begin with.

Brotli compresses my files way better, but it's doing it way slower. Anyway, universal statement "zstd is better" is not valid.

On max compression "--ultra -22", zstd is likely to be 2-4% less dense (larger) on text alike input. While taking over 2x times times to compress. Decompression is also much faster, usually over 2x.

I have not tried using a dictionary for zstd.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#108

Post by atiedebee »

Thoreandan wrote: post #74
atiedebee wrote: post #33

Earlier quoted context omitted.

I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompr…

Does your source .pdf material have FlateDecode'd chunks or did you fully uncompress it?

I wasn't sure. I just went in with the (probably faulty) assumption that if it compresses to less than 90% of the original size that it had enough "non-randomness" to compare compression performance.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#109

Post by atiedebee »

atiedebee wrote: post #33
dist-epoch wrote: post #25

Earlier quoted context omitted.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompr…

Ran the tests again with some more files, this time decompressing the pdf in advance. I picked some widely available PDFs to make the experiment reproducable.

  file            | raw         | zstd       (%)      | brotli     (%)     |
  gawk.pdf        | 8.068.092   | 1.437.529  (17.8%)  | 1.376.106  (17.1%) |
  shannon.pdf     | 335.009     | 68.739     (20.5%)  | 65.978     (19.6%) |
  attention.pdf   | 24.742.418  | 367.367    (1.4%)   | 362.578    (1.4%)  |
  learnopengl.pdf | 253.041.425 | 37.756.229 (14.9%)  | 35.223.532 (13.9%) |
For learnopengl.pdf I also tested the decompression performance, since it is such a large file, and got the following (less surprising) results using 'perf stat -r 5':

  zstd:   0.4532 +- 0.0216 seconds time elapsed  ( +-  4.77% )
  brotli: 0.7641 +- 0.0242 seconds time elapsed  ( +-  3.17% )
The conclusion seems to be consistent with what brotli's authors have said: brotli achieves slightly better compression, at the cost of a little over half the decompression speed.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#110

Post by tracker1 »

mort96 wrote: post #94
mort96 wrote: post #43

Earlier quoted context omitted.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I did my own testing where Brotli also ended up better than ZSTD: https://news.ycombinator.com/item?id=46722044 Results by compression type across 55 PDFs: +------…

Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size. Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size): +---------+---------+--------+--------+--------+ | none | zstd | xz | gzip | brotli…

Worth considering the compress/decompress overhead, which is also lower in brotli than zstd from my understanding.

Also, worth testing zopfli since it's decompression is gzip compatible.

Post reply on HN