Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QOI – The “Quite OK Image Format” for fast, lossless image compression

github.com

Post reply on HN
Print view View original

51–60 of 105 posts

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#51

Post by ErikCorry »

sreekotay wrote: post #10

If QOI is interesting because of speed, you might take a look at fpng, a recent/actively developed png reader/writer that is achieving comparable speed/compression to QOI, while staying png compliant. https://github.com/richgel999/fpng Disclaimer: have not actively tried either.

Funnily enough it's just a few days since I did some similar code to support writing PNGs from a small embedded device. In this case the full deflate algorithm seemed like overkill in memory and CPU requirement, and most of the images were probably going to be served over a LAN anyway. https://twitter.com/toitpkg/status/1471986776357097475

https://github.com/toitlang/toit/commit/65c6c1bd7138f9ebced4... It's not as highly optimized as this effort though, and it just uses the standard huffman table that is built into deflate, rather than a static-but-custom one.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#52

Post by ErikCorry »

jws wrote: post #21
sreekotay wrote: post #10

If QOI is interesting because of speed, you might take a look at fpng, a recent/actively developed png reader/writer that is achieving comparable speed/compression to QOI, while staying png compliant. https://github.com/richgel999/fpng Disclaimer: have not actively tried either.

I find it interesting that QOI avoids any kind of Huffman style coding. Huffman encoding lets you store frequently used values in fewer bits than rarely occurring values, but the cost of a naïve implementation is a branch on every encoded bit. You can mitigate this by making a state machine keyed by "accumulated prefix bits" and as many bits as you want to process in a whack, these tables will blow out your L1 data c…

I would think that you could use a hybrid approach where you have a table that is perhaps 9 or 10 bits wide and covers many of the more common codes, which will by definition be more common. Should be small enough to fit in the cache. Then do something slower for the very long codes. This way you avoid difficult branches most of the time.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#53

Post by ErikCorry »

midjji wrote: post #45
FullyFunctional wrote: post #30

Since we are rehashing this for the 3rd (4th?) time, I'll repeat mine (and apparently many others) key critique: there is no thought at all to enabling parallel decoding, be it, thread-parallel or SIMD (or both). That makes it very much a past millennium style format that will age very poorly. At the very least, break it into chunks and add an offset directory header. I'm sure one could do something much better, but…

A surprisingly good image format is to use a per line, or block, ar encoder, then compress the result with gzip on a low setting. It parallelizes very nicely, and beats png for encode, decode, and is trivial to implement.

You should be able to do the same technique, but in a PNG-compatible format.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#55

Post by junon »

petitg1987 wrote: post #54

I just implemented this format in my game engine and the performances are crazy: images loading is 3.2 times faster (compared to png) and 40 times faster to generate game screenshot!

And the size difference? mapping in a raw pixel data binary is infinitely faster than any image encoding, but takes up the most space of course.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#57

Post by petitg1987 »

junon wrote: post #55
petitg1987 wrote: post #54

I just implemented this format in my game engine and the performances are crazy: images loading is 3.2 times faster (compared to png) and 40 times faster to generate game screenshot!

And the size difference? mapping in a raw pixel data binary is infinitely faster than any image encoding, but takes up the most space of course.

The generated screenshots are lighter (about 5%). However, the resource images in QOI format that I load are in average a little bigger (about 5% and sometimes until 35%). I guess it is not the perfect solution for AAA games which already use more than 30go nowadays.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#58

Post by wombatmobile »

phonon wrote: post #41
wombatmobile wrote: post #31

Earlier quoted context omitted.

I'd also like to know what's the best (or any) lossless audio compression process/tools. My application is to send audio (podcast recordings) to a remote audio engineer friend who will do the post processing, then round trip it to me to complete the editing. Wav is so big it makes a 1 hr podcast a difficult proposition. MP3 is unsuitable because compression introduces too many artefacts the quality suffers unacceptab…

1 hour of CD quality mono FLAC encoded is about 100-150 MB. Is that small enough?

Well, I'm using a Rodecaster in multi-channel mode with 3 mics so an hour is more like 450MB.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#59

Post by chrismorgan »

jws wrote: post #21
sreekotay wrote: post #10

If QOI is interesting because of speed, you might take a look at fpng, a recent/actively developed png reader/writer that is achieving comparable speed/compression to QOI, while staying png compliant. https://github.com/richgel999/fpng Disclaimer: have not actively tried either.

I find it interesting that QOI avoids any kind of Huffman style coding. Huffman encoding lets you store frequently used values in fewer bits than rarely occurring values, but the cost of a naïve implementation is a branch on every encoded bit. You can mitigate this by making a state machine keyed by "accumulated prefix bits" and as many bits as you want to process in a whack, these tables will blow out your L1 data c…

Meta: ␄ (https://en.wikipedia.org/wiki/End-of-Transmission_character) isn’t the right control character when footnotes follow; ␃ (https://en.wikipedia.org/wiki/End-of-Text_character) is a better fit, and ␌ (https://en.wikipedia.org/wiki/Form_feed) would be a decent choice too.

(I write comments with footnotes in the same style as you, but use “—⁂—” as the separator, via Compose+h+r (name from the HTML tag horizontal rule). Good fun being able to use Compose+E+O+T, Compose+E+T+X and Compose+F+F in this comment; I added the full set to my .XCompose years ago.)

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#60

Post by midjji »

ErikCorry wrote: post #53
midjji wrote: post #45

Earlier quoted context omitted.

A surprisingly good image format is to use a per line, or block, ar encoder, then compress the result with gzip on a low setting. It parallelizes very nicely, and beats png for encode, decode, and is trivial to implement.

You should be able to do the same technique, but in a PNG-compatible format.

I'm really not a fan of using the generally unsupported aspects of the file formats. Like sure, png supports alternate compression schemes which in combination with the fully generic datablocks system which I think technically accidentally makes it Turing complete. But its not like any reader ever supports it fully, hell most readers dont support showing the other images in a png file. Its also quite common that palettes aren't properly supported, in particular for uncommon combination. Palettes themselves are also not truly sufficient for what they are supposed to do, as that would require generic scalar to scalar functions. Its like this with every damned old image format. People try to make the format to end all formats and end up accidentally inventing really shitty programming languages with terrible separation of concerns which are terribly supported but work as people only ever use them for single image data.

Another stranger mistake is the use of generic compression a part of the format, its much better to leave that up to the filesystem or stream. I dont really understand why this was ever a good idea, but it certainly hasn't been one for decades.

The more recent blob formats people have developed aren't much better, they still confuse the specification of a single blob with the specification of the container format, and try to be convenient and fully generic at once. If people actually wanted a fully generic dataformat and accepted that this requires a programming language, just let that be python, instead of inventing some shitty new one...

Post reply on HN