Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

pdfa.org

Post reply on HN
Print view View original

91–100 of 116 posts

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#91

Post by nxobject »

mmooss wrote: post #90
ericpauley wrote: post #7

Some real cognitive dissonance in this article… “The PDF Association operates under a strict principle—any new feature must work seamlessly with existing readers” followed by introducing compression as a breaking change in the same paragraph. All this for brotli… on a read-many format like pdf zstd’s decompression speed is a much better fit.

Note the language: "You're not creating broken files—you're creating files that are ahead of their time." Imagine a sales meeting where someone pitched that to you. They have to be joking, right? I have no objection to adding Brotli, but I hope they take the compatability more seriously. You may need readers to deploy it for a long time - ten years? - before you deploy it in PDF creation tools.

(sarcasm warning...)

You're absolutely right! It's not just an inaccurate slogan—it's a patronizing use of artificial intelligence. What you're describing is not just true, it's precise.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#92

Post by mort96 »

terrelln wrote: post #88
mort96 wrote: post #87

Earlier quoted context omitted.

Interesting! It doesn't seem to be only about overwriting, I can be in a directory without any .zst files and run the command to compress 55 files in parallel and it's still 45M according to 'du -h'. But you're right, 'wc -c' shows 38809999 bytes regardless of whether 'du -h' shows 45M after a parallel compression or 38M after a sequential compression. My mental model of 'du' was basically that it gives a size accura…

Yeah, it isn't quite that simple. E.g. `/bin/ksh` reports 1.4MB, but it is actually 2.4MB. Initially, I thought it was because the file was sparse, but there are only 493KB of zeros. So something else is going on. Perhaps some filesystem-level blocks are deduped from other files? Or APFS has transparent compression? I'm not sure. It does still seem odd that APFS is reporting a significantly larger disk-size for these…

Ross Burton on Mastodon suggests that it might be deduplication; when writing sequentially, later files can re-use blocks from earlier files, while that isn't the case as much when writing sequentially. That seems plausible enough to me.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#93

Post by gcr »

Bolwin wrote: post #82
gcr wrote: post #80

If we're making breaking changes to PDFs, I'd love if the committee added a modern image format like JPEG-XL. In my experience, most disk usage of PDFs comes from images, not streams. I keep a bunch of comics in PDF but JPEG-XL is by far the best way to enjoy them in terms of disk space.

Odd you should say that, as that's exactly what they've been discussing

No it's not. This article is about proposing Brotli as another possible '/Filter' for stream objects, like content streams (page drawing commands). Images are streams too, but unless you mean compressing raw pixel bytes in Brotli, there's no mention of a JPEG-XL or WEBP filter.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#94

Post by mort96 »

mort96 wrote: post #43
atiedebee wrote: post #33

Earlier quoted context omitted.

I thought the same, so I ran brotli and zstd on some PDFs I had laying around. brotli 1.0.7 args: -q 11 -w 24 zstd v1.5.0 args: --ultra -22 --long=31 | Original | zstd | brotli RandomBook.pdf | 15M | 4.6M | 4.5M Invoice.pdf | 19.3K | 16.3K | 16.1K I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompr…

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I did my own testing where Brotli also ended up better than ZSTD: https://news.ycombinator.com/item?id=46722044 Results by compression type across 55 PDFs: +------…

Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size.

Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size):

    +---------+---------+--------+--------+--------+
    |  none   |  zstd   |   xz   |  gzip  | brotli |
    +---------|---------|--------|--------|--------|
    | 47.81M  | 37.92M  | 37.96M | 38.80M | 37.06M |
    +---------+---------+--------+--------+--------+
These numbers are much more impressive. Still, Brotli has a slight edge.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#95

Post by gcr »

mort96 wrote: post #41
HackerThemAll wrote: post #27

Earlier quoted context omitted.

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

If you're worried about double-compression of image data, you can uncompress all images by using qpdf:

    qpdf --stream-data=uncompress in.pdf out.pdf
The resulting file should compress better with zstd.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#96

Post by mort96 »

mort96 wrote: post #41
HackerThemAll wrote: post #27

Earlier quoted context omitted.

I think this was the main reason (from the linked article) LOL: "Brotli is a compression algorithm developed by Google." They have no idea about Zstandard nor ANS/FSE comparing it with LZ77. Sheer incompetence.

EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158 I just took all PDFs I had in my downloads folder (55, totaling 47M). These are invoices, data sheets, employment contracts, schematics, research reports, a bunch…

Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size.

Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size):

    +---------+---------+--------+--------+--------+
    |  none   |  zstd   |   xz   |  gzip  | brotli |
    +---------|---------|--------|--------|--------|
    | 47.81M  | 37.92M  | 37.96M | 38.80M | 37.06M |
    +---------+---------+--------+--------+--------+
These numbers are much more impressive. Still, Brotli has a slight edge.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#97

Post by mort96 »

mort96 wrote: post #92
terrelln wrote: post #88

Earlier quoted context omitted.

Yeah, it isn't quite that simple. E.g. `/bin/ksh` reports 1.4MB, but it is actually 2.4MB. Initially, I thought it was because the file was sparse, but there are only 493KB of zeros. So something else is going on. Perhaps some filesystem-level blocks are deduped from other files? Or APFS has transparent compression? I'm not sure. It does still seem odd that APFS is reporting a significantly larger disk-size for these…

Ross Burton on Mastodon suggests that it might be deduplication; when writing sequentially, later files can re-use blocks from earlier files, while that isn't the case as much when writing sequentially. That seems plausible enough to me.

I've concluded that this can't be the reason. It'd only result in an error where the size reported by 'du' is smaller than the apparent size (aka number of bytes reported by 'wc -c') of the file. What we see here is that the size reported by 'du' is almost twice as large as the number of bytes. That can't be the result of dedpulication.

I'll chalk it up to "some APFS weirdness".

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#98

Post by superkuh »

kayodelycaon wrote: post #65
superkuh wrote: post #42

This is nice, but PDF jumped the shark already. It's no longer a document format that always looks the same everywhere. The inclusion of "Dynamic XFA (XML Form Architecture) PDF" in the spec made it so PDF is an unreliable format. The aformentioned is a PDF without content that pulls down all it's content from the web. It even still, ostensibly, supports Flash (swf) animations. In practice these "PDF"s are just empty…

Fortunately, XFA is deprecated. I haven’t seen one of those for a very long time.

Maybe in spec, but the damage is done and persists.

The (USA) Wisconsin Dept. of Natural Resources has nearly all their regulation PDFs as these XFA non-pdfs that I cannot read. So I cannot know the regulations. My emails about this topic (to multiple addresses over many years a dozen times) have gone unanswered.

If Acrobat supports it it doesn't matter what the spec says. Until Adobe drops XFA from Acrobat and forces these extremely silly people to stop, PDF is no longer PDF.

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#99

Post by NoahZuniga »

gcr wrote: post #93
Bolwin wrote: post #82

Earlier quoted context omitted.

Odd you should say that, as that's exactly what they've been discussing

No it's not. This article is about proposing Brotli as another possible '/Filter' for stream objects, like content streams (page drawing commands). Images are streams too, but unless you mean compressing raw pixel bytes in Brotli, there's no mention of a JPEG-XL or WEBP filter.

well, not mentioned in this specific article. But JPEG-XL support is something they're working on [1].

[1]: https://pdfa.org/wp-content/uploads/2025/10/PDFDays2025-Brea...

Re: ISO PDF spec is getting Brotli – ~20 % smaller documents with no quality loss

#100

Post by lmz »

eru wrote: post #36
ndriscoll wrote: post #30

What is the point of using a generic compression algorithm in a file format? Does this actually get you much over turning on filesystem and transport compression, which can transparently swap the generic algorithm (e.g. my files are already all zstd compressed. HTTP can already negotiate brotli or zstd)? If it's not tuned to the application, it seems like it's better to leave it uncompressed and let the user decide w…

Well, if sanity had prevailed, we would have likely stuck to .ps.gz (or you favourite compression format), instead of ending up with PDF. Though we might still want to restrict the subset of PostScript that we allow. The full language might be a bit too general to take from untrusted third parties.

Compression filters are in PostScript.
Post reply on HN