Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

pdfa.org

Post reply on HN
Print view View original

81–90 of 116 posts

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#81

Post by wizzwizz4 »

Someone wrote: post #40
ndriscoll wrote: post #30

What is the point of using a generic compression algorithm in a file format? Does this actually get you much over turning on filesystem and transport compression, which can transparently swap the generic algorithm (e.g. my files are already all zstd compressed. HTTP can already negotiate brotli or zstd)? If it's not tuned to the application, it seems like it's better to leave it uncompressed and let the user decide w…

- inside the file, the compressor can be varied according to the file content. For example, images can use jpeg, but that isn’t useful for compressing text - when jumping from page to page, you won’t have to decompress the entire file

> inside the file, the compressor can be varied according to the file content. For example, images can use jpeg, but that isn’t useful for compressing text

Okay, so we make a compressed container format that can perform such shenanigans, for the same amount of back-compat issues as extending PDF in this way.

> when jumping from page to page, you won’t have to decompress the entire file

This is already a thing with any compression format that supports quasi-random access, which is most of them. The answers to https://stackoverflow.com/q/429987/5223757 discuss a wide variety of tools for producing (and seeking into) such files, which can be read normally by tools not familiar with the conventions in use.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#82

Post by Bolwin »

gcr wrote: post #80

If we're making breaking changes to PDFs, I'd love if the committee added a modern image format like JPEG-XL. In my experience, most disk usage of PDFs comes from images, not streams. I keep a bunch of comics in PDF but JPEG-XL is by far the best way to enjoy them in terms of disk space.

Odd you should say that, as that's exactly what they've been discussing

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#83

Post by wizzwizz4 »

jsnell wrote: post #67
LoganDark wrote: post #44

Earlier quoted context omitted.

I love when I perform all the due diligence tasks. You just can't counter that. Yes but, they did all the due diligence tasks. They considered all the factors. Every one. Think you have one they didn't consider? Nope.

But they didn't write "all". They wrote "other", which absolutely does not imply full coverage. Maybe read things a bit more carefully before going all out on the snide comments?

In fact, they wrote "reviewing […] other due diligence tasks", which doesn't imply any coverage! This close, literal reading is an appropriate – nay, the only appropriate – way to draw conclusions about the degree of responsibility exhibited by the custodians of a living standard. By corollary, any criticism of this form could be rebuffed by appeal to a sufficiently-carefully-written press release.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#84

Post by terrelln »

mort96 wrote: post #78
terrelln wrote: post #75

Earlier quoted context omitted.

Yeah, `--adaptive` will enable adaptive compression, but it isn't enabled by default, so shouldn't apply here. But even with `--adaptive`, after compressing each block of 128KB of data, zstd checks that the output size is So it is very central to zstd that it will never emit a block that is larger than 128KB+3B. I will try to reproduce, but I suspect that there is something unrelated to zstd going on. What version of…

'zstd --version' reports: "** Zstandard CLI (64-bit) v1.5.7, by Yann Collet **". This is zstd installed through Homebrew on macOS 26 on an M1 Pro laptop. Also of interest, I was able to reproduce this with a random binary I had in /bin: https://floss.social/@mort/115940378643840495 I was completely unable to reproduce it on my Linux desktop though: https://floss.social/@mort/115940627269799738

I've figured out the issue. Use `wc -c` instead of `du`.

I can repro on my Mac with these steps with either `zstd` or `gzip`:

    $ rm -f ksh.zst
    $ zstd  ksh.zst
    $ du -h ksh.zst
    1.2M ksh.zst
    $ wc -c ksh.zst
     1240701 ksh.zst
    $ zstd  ksh.zst
    $ du -h ksh.zst
    2.0M ksh.zst
    $ wc -c ksh.zst
     1240701 ksh.zst
    
    $ rm -f ksh.gz
    $ gzip  ksh.gz
    $ du -h ksh.gz
    1.2M ksh.gz
    $ wc -c ksh.gz
     1246815 ksh.gz
    $ gzip  ksh.gz
    $ du -h ksh.gz
    2.1M ksh.gz
    $ wc -c ksh.gz
     1246815 ksh.gz
When a file is overwritten, the on-disk size is bigger. I don't know why. But you must have ran zstd's benchmark twice, and every other compressor's benchmark once.

I'm a zstd developer, so I have a vested interest in accurate benchmarks, and finding & fixing issues :)

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#85

Post by jhealy »

whinvik wrote: post #57

I am often frustrated by PDF issues such as how complicated it is to create one. But reading the article I realized PDFs have become ubiquitous because of its insistence on backwards compatibility. Maybe for some things it's good to move this slow.

The article is wrong, the PDF spec has introduced breaking changes plenty of times. It’s done slowly and conservatively though, particularly now that the format is an ISO spec.

The PDF format is versioned, and in the past new versions have introduced things like new types of encryption. It’s quite probable that a v1.7 compliant PDF won’t open on a reader app written when v1.3 was the latest standard.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#86

Post by stonogo »

adzm wrote: post #17
xxs wrote: post #9

Earlier quoted context omitted.

yup, zstd is better. Overall use zstd for pretty much anything that can benefit from a general purpose compression. It's a beyond excellent library, tool, and an algorithm (set of). Brotli w/o a custom dictionary is a weird choice to begin with.

Brotli makes a bit of sense considering this is a static asset; it compresses somewhat more than zstd. This is why brotli is pretty ubiquitous for precompressed static assets on the Web. That said, I personally prefer zstd as well, it's been a great general use lib.

brotli is ubiquitous because Google recommends it. While Deflate definitely sucks and is old, Google ships brotli in Chrome, and since Chrome is the de facto default platform nowadays, I'd imagine it was chosen because it was the lowest-effort lift.

Nevertheless, I expect this to be JBIG2 all over again: almost nobody will use this because we've got decades of devices and software in the wild that can't, and 20% filesize savings is pointless if your destination can't read the damn thing.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#87

Post by mort96 »

terrelln wrote: post #84
mort96 wrote: post #78

Earlier quoted context omitted.

'zstd --version' reports: "** Zstandard CLI (64-bit) v1.5.7, by Yann Collet **". This is zstd installed through Homebrew on macOS 26 on an M1 Pro laptop. Also of interest, I was able to reproduce this with a random binary I had in /bin: https://floss.social/@mort/115940378643840495 I was completely unable to reproduce it on my Linux desktop though: https://floss.social/@mort/115940627269799738

I've figured out the issue. Use `wc -c` instead of `du`. I can repro on my Mac with these steps with either `zstd` or `gzip`: $ rm -f ksh.zst $ zstd ksh.zst $ du -h ksh.zst 1.2M ksh.zst $ wc -c ksh.zst 1240701 ksh.zst $ zstd ksh.zst $ du -h ksh.zst 2.0M ksh.zst $ wc -c ksh.zst 1240701 ksh.zst $ rm -f ksh.gz $ gzip ksh.gz $ du -h ksh.gz 1.2M ksh.gz $ wc -c ksh.gz 1246815 ksh.gz $ gzip ksh.gz $ du -h ksh.gz 2.1M ksh.gz…

Interesting!

It doesn't seem to be only about overwriting, I can be in a directory without any .zst files and run the command to compress 55 files in parallel and it's still 45M according to 'du -h'. But you're right, 'wc -c' shows 38809999 bytes regardless of whether 'du -h' shows 45M after a parallel compression or 38M after a sequential compression.

My mental model of 'du' was basically that it gives a size accurate to the nearest 4k block, which is usually accurate enough. Seems I have to reconsider. Too bad there's no standard alternative which has the interface of 'du' but with byte-accurate file sizes...

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#88

Post by terrelln »

mort96 wrote: post #87
terrelln wrote: post #84

Earlier quoted context omitted.

I've figured out the issue. Use `wc -c` instead of `du`. I can repro on my Mac with these steps with either `zstd` or `gzip`: $ rm -f ksh.zst $ zstd ksh.zst $ du -h ksh.zst 1.2M ksh.zst $ wc -c ksh.zst 1240701 ksh.zst $ zstd ksh.zst $ du -h ksh.zst 2.0M ksh.zst $ wc -c ksh.zst 1240701 ksh.zst $ rm -f ksh.gz $ gzip ksh.gz $ du -h ksh.gz 1.2M ksh.gz $ wc -c ksh.gz 1246815 ksh.gz $ gzip ksh.gz $ du -h ksh.gz 2.1M ksh.gz…

Interesting! It doesn't seem to be only about overwriting, I can be in a directory without any .zst files and run the command to compress 55 files in parallel and it's still 45M according to 'du -h'. But you're right, 'wc -c' shows 38809999 bytes regardless of whether 'du -h' shows 45M after a parallel compression or 38M after a sequential compression. My mental model of 'du' was basically that it gives a size accura…

Yeah, it isn't quite that simple. E.g. `/bin/ksh` reports 1.4MB, but it is actually 2.4MB. Initially, I thought it was because the file was sparse, but there are only 493KB of zeros. So something else is going on. Perhaps some filesystem-level blocks are deduped from other files? Or APFS has transparent compression? I'm not sure.

It does still seem odd that APFS is reporting a significantly larger disk-size for these files. I'm not sure why that would ever be the case, unless there is something like deferred cleanup work.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#89

Post by cess11 »

noname120 wrote: post #49
cess11 wrote: post #10

'Your PDF:s will open slower because we decided that the CDN providers are more important than you'. If size was important to users then it wouldn't be so common that systems providers crap out huge PDF files consisting mainly of layout junk 'sophistication' with rounded borders and whatnot. The PDF/A stuff I've built stays under 1 MB for hundreds of pages of information, because it's text placed in a typographically…

Ridiculous statement. CDN providers can already use filesystem compression and standard HTTP Accept-Encoding compression for transfers (which includes brotli by the way). This ISO provides virtually no benefit to them

This reasoning comes from TFA.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#90

Post by mmooss »

ericpauley wrote: post #7

Some real cognitive dissonance in this article… “The PDF Association operates under a strict principle—any new feature must work seamlessly with existing readers” followed by introducing compression as a breaking change in the same paragraph. All this for brotli… on a read-many format like pdf zstd’s decompression speed is a much better fit.

Note the language: "You're not creating broken files—you're creating files that are ahead of their time."

Imagine a sales meeting where someone pitched that to you. They have to be joking, right?

I have no objection to adding Brotli, but I hope they take the compatability more seriously. You may need readers to deploy it for a long time - ten years? - before you deploy it in PDF creation tools.

Post reply on HN