Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QOI: Lossless Image Compression in O(n) Time

phoboslab.org

Post reply on HN
Print view View original

211–220 of 303 posts

Re: QOI: Lossless Image Compression in O(n) Time

#211

Post by dylan604 »

em3rgent0rdr wrote: post #204
jiggawatts wrote: post #23

I know you don't like "design by committee" and overly complex formats. Also, the compression algorithm itself is somewhat independent of the "container format" that it's used with. However, if I may make a suggestion: If you do end up rolling your own container format for any reason, always include a way to store the colour space of the image! Treating RGB images as arrays of bytes without further tags is like treat…

That metadata could be stored in the filename to keep it out of the actual file.

No, the filename is the worst place to put metadata. Nobody can control what will happen to a file name.

Re: QOI: Lossless Image Compression in O(n) Time

#212

Post by alkonaut »

jameshart wrote: post #162
cyber_kinetist wrote: post #47

Earlier quoted context omitted.

I think the author's intending to use this only as a data packing format (ex. for app/game development) and not a general-purpose exchange format. If all you want to do is pack image/texture data for your game then I totally get why they assume RGBA8 storage and leave out metadata such as color space information.

Right - this thread of people asking for colorspaces, dpi data, arbitrary metadata collections… all are exhibiting precisely the ‘expert group’ committee mindset that this format is supposed to avoid. If you want all that, there are formats galore that support it! The point is more: where are the formats for people who don’t need all that? Here is one. Please don’t try to ruin it.

You don’t need to support a ton of complex stuff, but if an rgb image format doesn’t then it should just have a version marker and the spec can simply say “the rgb data represents sRGB colors”.

Just like e.g. some compilers don’t support a lot of source file encodings but their non-support consists of clearly stating that a source file is UTF-8.

Note that this requirement is only for an image exchange format. A raw-er image format used internally in e.g a game doesn’t need it as badly.

But in an exchange format every user needs to know whether the image data is sRGB or Adobe. Because the reader didn’t encode the data.

Re: QOI: Lossless Image Compression in O(n) Time

#213

Post by Dylan16807 »

lstamour wrote: post #139
demute wrote: post #131

Earlier quoted context omitted.

CIE XYZ is complete.

Isn’t that kind of like saying Real numbers are complete? Yes, it’s true, but we don’t have an infinite level of precision for each colour. So we have to, practically speaking, limit how many colours can be specified to some reasonable (and arbitrary) number, e.g. 10 million or 1 billion or more (in terms of number of colours in a colour space) and representation in digital bits. Also, since I conflated the two a bit…

> Isn’t that kind of like saying Real numbers are complete?

It's more like saying the range from 0..2 is complete so don't use an encoding that caps out at 0.8 or 1.1

The difference between all visible colors and a more constrained color space is only a factor of two or three. Less than half a bit per channel.

Re: QOI: Lossless Image Compression in O(n) Time

#214

Post by Dylan16807 »

kortex wrote: post #184
demute wrote: post #131

Earlier quoted context omitted.

CIE XYZ is complete.

CIE XYZ has imaginary colors, though. I guess when I said "no color space is complete" I meant "no color space is faithfully bijective to nominative human visual sensory response." Because such a thing is basically impossible (or insanely nonlinear and unwieldy). Also these color systems all or almost all assume a "norminal" (to steal a word from Everyday Astronaut, normal+nominal) average human vision system. If you…

Why does it matter if it's bijective?

Re: QOI: Lossless Image Compression in O(n) Time

#215

Post by Dylan16807 »

pixel_fcker wrote: post #191
formerly_proven wrote: post #182

Earlier quoted context omitted.

> An absolute color space exists. It's called XYZ. We could use it. Some image formats support it. XYZ is really annoying to work in though. ACES is a pragmatic solution here, quite literally: https://blog.frame.io/wp-content/uploads/2019/09/ACES-APO.jp... Okay, fair enough two out of three primaries are imaginary colors, but nobody said you have to use the whole color space when your processing pipeline is meant to…

ACES (AP0) is an archive format for when you want to be absolutely sure you’re not clipping any colours. As a working space it’s terrible and AP1 should be preferred, either as ACEScg or ACEScc

AP1 is almost the same as rec.2020? I'm not sure what you mean by 'working space' but if preserving all colors was a primary or even secondary goal I definitely wouldn't cut myself down that far.

Re: QOI: Lossless Image Compression in O(n) Time

#216

Post by cornstalks »

spc476 wrote: post #195
cornstalks wrote: post #178

Earlier quoted context omitted.

> When I say (255, 0, 0), I expect the display to show me its best approximation of red. Which red? There isn't a single, true "red". And different materials produce different reds (my work's Sony BVM-X300's reddest red is going to look way different than your monitor's red). Not all displays use only RGB color primaries, too. For example, at Dolby's office in San Francisco they have a projector in their theater that…

And the red in my left eye isn't the same as the red in my right eye. Yes, when the lighting conditions are just right , I see different hues out of each eye [1]. I have to wonder how one would (or should?) correct for that in an image file format. [1] I think from a stupid thing I did as a kid.

I think the best we can do with file-level metadata is account for only objective metrics. Subjective differences are best handled outside of the file metadata. The user can tune their display settings to match their preferences. This allows for correcting for physiological differences and stylistic preferences. If all files had correct metadata (for objective metrics), it would be a lot easier for an end user to tune the system to their liking because their end results would be consistent with their preferences.

Re: QOI: Lossless Image Compression in O(n) Time

#217

Post by Animats »

Lossless avoids all those issues around human perception. Also, if you're storing game assets, some of what you're storing as images is not really imagery. It may be normals or height fields, which lossy image compression messes up.

The code is interesting, in that it iterates over the output array, not the input. This helps prevent write buffer overflows. It has read buffer overflows, though. You can craft a compressed file which, when decompressed, will contain info from elsewhere in memory.

Re: QOI: Lossless Image Compression in O(n) Time

#218

Post by Const-me »

jiggawatts wrote: post #20
junon wrote: post #3

To answer the author's question: No, SIMD would not help with this. This is neat. I wonder if the author would be willing to write a Kaitai Struct definition for it. Something else interesting: QOI is 1,2,2 letters off from PNG. I'm quite certain this is an accident but it's interesting nonetheless.

I suspect SIMD would help with the encoding. The lookup table is small enough to fit into 8 AVX2 registers, so instead of hashing, you could use direct lookup, which would improve compression ratio further (a little bit).

> The lookup table is small enough to fit into 8 AVX2 registers

Indeed.

> so instead of hashing, you could use direct lookup

However, I don’t think that part gonna work. See the code using that table: https://github.com/phoboslab/qoi/blob/master/qoi.h#L324-L328

SIMD registers aren’t indexable (at least not on AMD64), the register needs to be known to the compiler.

Lanes within each register aren’t indexable either. The insert and extract instructions are encoding lane index in the code. There’re workarounds for this one, like abusing vpshufb or vpermd, but with that amount of overhead I doubt SIMD will deliver any profit at all.

Re: QOI: Lossless Image Compression in O(n) Time

#219

Post by romwell »

kaetemi wrote: post #163
sam0x17 wrote: post #159

Earlier quoted context omitted.

> We're stuck with images stored in the display profile of old CRTs by default, because that was the most practical option at the time. The analogue in photography is photographs lose color clarity and fade as they age. Why should I care if this is the case in images as display technology evolves when this is already a problem with every physical medium? > For correct editing operations, and correct display. Sadly, i…

I get what you mean, but... the color profile of the image is the profile of the display where the image can be displayed without adjustment. Think of it as the display profile of the guy that sent you the image. The magic math transforms the image to your display profile. That means it will look exactly the same on both displays. If they both have a correct display profile. If your green is going pink, then either y…

>That means it will look exactly the same on both displays. If they both have a correct display profile

We can continue believing that Santa exists, or we can accept that effectively nobody has correct color profiles, and doesn't care either.

It's nice metadata you have there, would be a shame if I applied night mode to it at 6PM.

> also, you can actually calibrate consumer hardware

...with professional hardware that costs more than the one you have at hand.

Again, pretty much everyone will just tweak their monitor/printer settings until they get results that look OK.

>display and print will look exactly the same

Until you turn off the lights. Or until you turn on the lights (what color temperature is you light?). Or until the wind blows, moving that cloud over the sun, changing light temperature.

Or — wait for it — actually, that's all, while you've been waiting the sun has set.

The point being, all we can shoot for is FF0000 being what would be "red" for most, 00FF00 being "green", and 0000FF being "blue" — and then accept the "fade" of the digital image from one screen to another as a fact of life.

So, learn how to stop worrying and love unspecified RGB colorpsaces. Magenta only exists in your brain anyway [1]

Side note: did you ever think about how your recorded music isn't adjusted for the frequency response of the loudspeakers, and half the people will listen to it with the "BASS MEGABOOST" feature on anyway?

That's why the art of mastering exists. It's accepting the reality, and putting work into making it work with uncalibrated, crappy hardware — as well as hi-fi gear.

PS: have fun calibrating your vision to make sure that you don't see green where I see red

[1]https://medium.com/swlh/magenta-the-color-that-doesnt-exist-...

Re: QOI: Lossless Image Compression in O(n) Time

#220

Post by hn_acc_2 »

bluedino wrote: post #208
rytill wrote: post #138

Earlier quoted context omitted.

Why use two files instead of one?

What if you want to use the functions in other C files? What if you want to compile them all separately?

1) Include it wherever it's used. The convention is to use the preprocessor[1] so it only appears once in the final binary

2) Most developers just want to compile to a single binary. Any developer who for some reason needs separately compiled objects should be able to quickly achieve that from a single-file library

[1] https://github.com/phoboslab/qoi/blob/324c2243b2fb35f741a88d...

Post reply on HN