Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

pdfa.org

Post reply on HN
Print view View original

51–60 of 116 posts

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#51

Post by mort96 »

noname120 wrote: post #47
mort96 wrote: post #41

Earlier quoted context omitted.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

Why not use a more widespread compression algorithm (e.g. gzip) considering that Brotli barely performs better at all? Sounds like a pain for portability

I'm not sold on the idea of adding compression to PDF at all, I'm not convinced that the space savings are worth breaking compatibility with older readers. Especially when you consider that you can just compress it in transit with e.g HTTP's 'Content-Encoding' without any special PDF reader support. (You can even use 'Content-Encoding: br' for brotli!)

If you do wanna change PDF backwards-incompatibly, I don't think there's a significant advantage to choosing gzip to be honest, both brotli and zstd are pretty widely available these days and should be fairly easy to vendor. But yeah, it's a slight advantage I guess. Though I would expect that there are other PDF data sets where brotli has a larger advantage compared to gzip.

But what I really don't get is all the calls to use zstd instead of brotli and treating the choise to use brotli instead of zstd as some form of Google conspiracy. (Is Facebook really better?)

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#52

Post by [deleted] »

noname120 wrote: post #50
mort96 wrote: post #41

Earlier quoted context omitted.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

Could you add compression and decompression speeds to your table?

[deleted]

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#53

Post by mort96 »

noname120 wrote: post #50
mort96 wrote: post #41

Earlier quoted context omitted.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

Could you add compression and decompression speeds to your table?

I just did some interactive shell loops and globs to compress everything and output CSV which I processed into an ASCII table, so I don't exactly have a pipeline I can modify and re-run the tests with compression speeds added ... but I can run some more interactive shell-glob-and-loop-based analysis to give you decompression speeds:

    ~/tmp/pdfbench $ hyperfine --warmup 2 \
    'for x in zst/*; do zstd -d >/dev/null /dev/null /dev/null /dev/null /dev/null /dev/null /dev/null /dev/null /dev/null /dev/null /dev/null /dev/null
As expected, xz is super slow. Gzip is fastest, zstd being somewhat slower, brotli slower again but still much faster than xz.

    +-------+-------+--------+-------+
    | gzip  | zstd  | brotli | xz    |
    +-------+-------+--------+-------+
    | 143ms | 165ms | 255ms  | 982ms |
    +-------+-------+--------+-------+
I honestly expected zstd to win here.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#54

Post by itsdesmond »

dist-epoch wrote: post #25
adzm wrote: post #17

Earlier quoted context omitted.

Brotli makes a bit of sense considering this is a static asset; it compresses somewhat more than zstd. This is why brotli is pretty ubiquitous for precompressed static assets on the Web. That said, I personally prefer zstd as well, it's been a great general use lib.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

> Pareto

I don’t think you’re using that correctly.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#55

Post by Proclus »

bhouston wrote: post #4

Are they using a custom dictionary with Brotli designed for PDFs? I am not sure if it would help or not, but it seems like one of those cases it may help? Something like this: https://developer.chrome.com/blog/shared-dictionary-compress... In my applications, in the area of 3D, I've been moving away from Brotli because it is just so slow for large files. I prefer zstd, because it is like 10x faster for both compressi…

It seems they're using the standard dictionary, which is utterly bizzare.

The standard Brotli dictionary bakes in a ton of assumptions about what the Web looked like in 2015, including not just which HTML tags were particularly common but also such things as which swear words were trendy.

It doesn't seem reasonable to think that PDFs have symbol probabilities remotely similar to the web corpus Google used to come up with that dictionary.

On top of that, it seems utterly daft to be baking that into a format which is expected to fit archival use cases and thus impose that 2015 dictionary on PDF readers for a century to come.

I too would strongly prefer that they use zstd.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#56

Post by mrspuratic »

atiedebee wrote: post #33
dist-epoch wrote: post #25

Earlier quoted context omitted.

You need to crank up zstd compression level. zstd is Pareto better than brotli - compresses better and faster

I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompr…

> I couldn't quickly find a way to decompress them

    pdftk in.pdf output out.pdf decompress

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#57

Post by whinvik »

I am often frustrated by PDF issues such as how complicated it is to create one.

But reading the article I realized PDFs have become ubiquitous because of its insistence on backwards compatibility. Maybe for some things it's good to move this slow.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#58

Post by bhouston »

Proclus wrote: post #55
bhouston wrote: post #4

Are they using a custom dictionary with Brotli designed for PDFs? I am not sure if it would help or not, but it seems like one of those cases it may help? Something like this: https://developer.chrome.com/blog/shared-dictionary-compress... In my applications, in the area of 3D, I've been moving away from Brotli because it is just so slow for large files. I prefer zstd, because it is like 10x faster for both compressi…

It seems they're using the standard dictionary, which is utterly bizzare. The standard Brotli dictionary bakes in a ton of assumptions about what the Web looked like in 2015, including not just which HTML tags were particularly common but also such things as which swear words were trendy. It doesn't seem reasonable to think that PDFs have symbol probabilities remotely similar to the web corpus Google used to come up…

BTW I've looked into custom dictionaries before for similar use cases and I suspect it would only offer like a 1% improvement or so for PDFs -- still good, but not a massive difference maker. The issue is that PDFs, like web pages, are incredibly repetitive in terms of their tags/structure. As such the custom dictionary only helps if the doc is really small, otherwise because of the repetitive nature, the self-inferred dictionary will resemble the custom dictionary after just a few blocks of PDF content.

The sole exception is if they are restarting the brotli stream for each page, and they are not sharing a dictionary, custom or inferred across the whole doc. Then the dictionary will have to be re-inferred on each page, and then a shared custom dictionary would make more sense.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#59

Post by terrelln »

mort96 wrote: post #41
HackerThemAll wrote: post #27

Earlier quoted context omitted.

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

> | 1.1M | 2.0M | 1.1M | 1.1M | 1.1M |

Something is going terribly wrong with `zstd` here, where it is reported to compress a file of 1.1MB to 2MB. Zstd should never grow the file size by more than a very small percent, like any compressor. Am I interpreting it correctly that you're doing something like `zstd -22 --ultra $FILE && wc -c $FILE.zst`?

If you can reproduce this behavior, can you please file an issue with the zstd version you are using, the commands used, and if possible the file producing this result.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#60

Post by terrelln »

mort96 wrote: post #53
noname120 wrote: post #50

Earlier quoted context omitted.

Could you add compression and decompression speeds to your table?

I just did some interactive shell loops and globs to compress everything and output CSV which I processed into an ASCII table, so I don't exactly have a pipeline I can modify and re-run the tests with compression speeds added ... but I can run some more interactive shell-glob-and-loop-based analysis to give you decompression speeds: ~/tmp/pdfbench $ hyperfine --warmup 2 \ 'for x in zst/*; do zstd -d >/dev/null /dev/n…

Zstd should not be slower than gzip to decompress here. Given that it has inflated the files to be bigger than the uncompressed data, it has to do more work to decompress. This seems like a bug, or somehow measuring the wrong thing, and not the expected behavior.
Post reply on HN