Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QOI – The “Quite OK Image Format” for fast, lossless image compression

github.com

Post reply on HN
Print view View original

21–30 of 105 posts

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#21

Post by jws »

sreekotay wrote: post #10

If QOI is interesting because of speed, you might take a look at fpng, a recent/actively developed png reader/writer that is achieving comparable speed/compression to QOI, while staying png compliant. https://github.com/richgel999/fpng Disclaimer: have not actively tried either.

I find it interesting that QOI avoids any kind of Huffman style coding.

Huffman encoding lets you store frequently used values in fewer bits than rarely occurring values, but the cost of a naïve implementation is a branch on every encoded bit. You can mitigate this by making a state machine keyed by "accumulated prefix bits" and as many bits as you want to process in a whack, these tables will blow out your L1 data cache and trash a lot of your L2 cache as well.¹

The "opcode" strategy in QOI is going to give you branches, but they appear nearly perfectly predictable for common image types², so that helps. It has a table of recent colors, but that is only of a few cache lines.

In all, it seems a better fit for the deep pipelines and wildly varying access speeds across cache and memory layers which we find today.

␄

¹ I don't think it ever made it into a paper, but in the mid-80s, when the best our Vax ethernet adapters could do was ~3Mbps I was getting about 10Mbps of decompressed 12 bit monochrome imagery out of a ~1.3MIP computer using this technique.

² I also wouldn't be surprised if this statement is false. It just seems that for continuous tone images one of RGBA, DIFF, or LUMA is going to win for any given region of a scan line.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#22

Post by 323 »

LeoPanthera wrote: post #11
zigzag312 wrote: post #8

Earlier quoted context omitted.

FLAC is limited to 24 bit depth. I was thinking of intermediate format suitable for use in DAWs and samplers that also supports floating point to avoid clipping.

24-bit integer and 32-bit float have the same dynamic range available, so you are not losing any fidelity. However, frankly, if you're working professionally with audio like that, the best solution is simply to have sufficient disk space available to work with raw audio. Use FLAC to compress the final product, when you are done.

All professional audio production software these days internally works with 32/64 bit floats. That's the native format, because it allows you to go above 0 dBFS (maximum level), as long as you go back below it at the end of the chain.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#23

Post by zigzag312 »

artiii wrote: post #16
zigzag312 wrote: post #8

Earlier quoted context omitted.

FLAC is limited to 24 bit depth. I was thinking of intermediate format suitable for use in DAWs and samplers that also supports floating point to avoid clipping.

check WavPack (32pcm, floats etc) but it's slower(not much) than flac, offering slighty beter compresion.

WavPack seems a bit too slow already. 3x slower decode compared to FLAC in this test https://stsaz.github.io/fmedia/audio-formats/

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#24

Post by zigzag312 »

323 wrote: post #20
zigzag312 wrote: post #3

Is there any open source audio compression format like that? Lossless and very fast. I haven't found any yet. EDIT: I'm thinking about a format that would be suitable as a replacement for uncompressed WAV files in DAWs. Rendered tracks often have large sections of silence and uncompressed WAVs have always seemed wasteful to me.

WavPack might fit the bill. It has decent software support. Not sure if DAWs can use it natively, they might unpack it to a temp folder. https://www.wavpack.com/

Reaper does. Unfortunately, WavPack has a bit too much performance overhead.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#25

Post by ReactiveJelly »

jqpabc123 wrote: post #5

Interesting format. It would be much more interesting if browsers supported it.

It's always gonna be chicken-and-egg for this, and browsers won't spend the time sandboxing and supporting a codec until it's already popular.

So this will probably see a JS / Webasm shim, and if that proves popular, Blink and Gecko will consider it.

The day might come soon when browsers just greenlight a webasm interface for codecs. "We'll put packets in through this function, and take frames out through this function, like ffmpeg. Other than that, you're running in a sandbox with X MB of RAM, Y seconds of CPU per frame, and no I/O. Anything you can accomplish within that, with user opt-in, is valid."

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#26

Post by 323 »

zigzag312 wrote: post #24
323 wrote: post #20

Earlier quoted context omitted.

WavPack might fit the bill. It has decent software support. Not sure if DAWs can use it natively, they might unpack it to a temp folder. https://www.wavpack.com/

Reaper does. Unfortunately, WavPack has a bit too much performance overhead.

It's a 20 year old format.

ZStandard is a very good compressor, with an especially fast decompressor. Maybe someone should try using this instead of zlib in an audio format (FLAC, WavPack, ...)

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#27

Post by dnautics »

jqpabc123 wrote: post #5

Interesting format. It would be much more interesting if browsers supported it.

Not sure what you're expecting given how old it is. Why not write a polyfill as an exercise for yourself? Convert it to png, then save as an image tag to a data url.

Here look some people adapted to ios in one hour faffing around on twitch: https://www.twitch.tv/videos/1241476768?tt_medium=mobile_web...

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#28

Post by jzwinck »

zigzag312 wrote: post #9
dmitrygr wrote: post #4

Earlier quoted context omitted.

gzip -1 is lossless and fast. It will somewhat compress pcm data :)

You would loose fast seeking ability with gzip. Or am I mistaken?

You can only seek within a gzip file if you write it with some number of Z_FULL_FLUSH points which are resumable. The command line gzip program does not support this, but it's easy using zlib. For example you might do a Z_FULL_FLUSH roughly every 50 MB of compressed data. Then you can seek to any byte in the file, search forward or backward for the flush marker, and decompress forward from there as much as you want. If your data is sorted or is a time series, it's easy to implement binary search this way. And standard gunzip etc will still be able to read the whole file as normal.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#29

Post by makapuf »

LeoPanthera wrote: post #7
zigzag312 wrote: post #3

Is there any open source audio compression format like that? Lossless and very fast. I haven't found any yet. EDIT: I'm thinking about a format that would be suitable as a replacement for uncompressed WAV files in DAWs. Rendered tracks often have large sections of silence and uncompressed WAVs have always seemed wasteful to me.

FLAC is always lossless, but has a variable compression ratio so you can trade compression for speed. Using the command line "flac" tool, "flac -0" is the fastest, "flac -8" is the slowest, but produces the smallest files. In my experience, 0-2 all produce roughly equivalent sized files, as do 4-8.

I tried passing stereo wavs in 2 x 16bits (4bytes) as rgba for qoi but I haven't been very successful.

Re: QOI – The “Quite OK Image Format” for fast, lossless image compression

#30

Post by FullyFunctional »

Since we are rehashing this for the 3rd (4th?) time, I'll repeat mine (and apparently many others) key critique: there is no thought at all to enabling parallel decoding, be it, thread-parallel or SIMD (or both). That makes it very much a past millennium style format that will age very poorly.

At the very least, break it into chunks and add an offset directory header. I'm sure one could do something much better, but it's a start.

EDIT: typo

Post reply on HN