Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Decoding AVIF: Deep dive with cats and imgproxy

evilmartians.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Decoding AVIF: Deep dive with cats and imgproxy

#11

Post by HALtheWise »

dylan604 wrote: post #2

"Commonly, three numbers are used to specify downsampling: the first is always 4, don’t ask me why" Wow. Someone going to this much detail on explaining how a video/image codec works, and cannot bother learning what the numbers of chroma subsampling mean? The first number represents the luminance.[0] Even if they know the first number represents luminance, the "don't ask me why" is just horrible on its own. The detai…

One thing I never understood is why _downsampling_ is the most efficient way to compress the data about chroma into fewer bits while maximizing perceptual accuracy. It really seems like for any given target bitrate for the chroma data, there should always be a more efficient compression scheme available than simply throwing out 3/4 of the pixels and running compression algorithms on the rest. Surely modern compression can do better with a continuous low pass filter or a adaptive compression scheme that focuses data on interesting edges or something? Maybe someone here can better explain the intuition for this. I'm similarly curious for resolution in general (i.e. why does 480p upsampled ever look better than 1080p at the same bitrate) but chroma seems like a good place to start.

Re: Decoding AVIF: Deep dive with cats and imgproxy

#12

Post by dylan604 »

HALtheWise wrote: post #11
dylan604 wrote: post #2

"Commonly, three numbers are used to specify downsampling: the first is always 4, don’t ask me why" Wow. Someone going to this much detail on explaining how a video/image codec works, and cannot bother learning what the numbers of chroma subsampling mean? The first number represents the luminance.[0] Even if they know the first number represents luminance, the "don't ask me why" is just horrible on its own. The detai…

One thing I never understood is why _downsampling_ is the most efficient way to compress the data about chroma into fewer bits while maximizing perceptual accuracy. It really seems like for any given target bitrate for the chroma data, there should always be a more efficient compression scheme available than simply throwing out 3/4 of the pixels and running compression algorithms on the rest. Surely modern compressio…

>Surely modern compression can do better

I "surely" look forward to your Show HN write up on your new compression algorithm. We've been iteratively getting better at compression for some time now. It seems like everytime it looks like we've wrung every bit out of DCT, someone comes up with some a little more clever. Wavelets looked promising, but never took off.

>why does 480p upsampled ever look better than 1080p at the same bitrate

That's a very vague question. Are you stating that you think 480p upsampled to 1080p at 1.5Mbps looks better than a source at 1080p at 1.5Mbps? I have a hard time believing this to be true.

To understand why the chroma is sub-sampled and not the luminance has to do with how the cones/rods in the eyes work. There's a lot of things you can get away with (or trick if you will) the brain in what it is seeing. Is it better to lose half the height or half the width? Is it better loose more red than green or blue?

Re: Decoding AVIF: Deep dive with cats and imgproxy

#13

Post by Scaevolus »

HALtheWise wrote: post #11
dylan604 wrote: post #2

"Commonly, three numbers are used to specify downsampling: the first is always 4, don’t ask me why" Wow. Someone going to this much detail on explaining how a video/image codec works, and cannot bother learning what the numbers of chroma subsampling mean? The first number represents the luminance.[0] Even if they know the first number represents luminance, the "don't ask me why" is just horrible on its own. The detai…

One thing I never understood is why _downsampling_ is the most efficient way to compress the data about chroma into fewer bits while maximizing perceptual accuracy. It really seems like for any given target bitrate for the chroma data, there should always be a more efficient compression scheme available than simply throwing out 3/4 of the pixels and running compression algorithms on the rest. Surely modern compressio…

JPEG XL doesn't perform chroma subsampling in its native color space of XYB. https://cloudinary.com/blog/how_jpeg_xl_compares_to_other_im...

Re: Decoding AVIF: Deep dive with cats and imgproxy

#14

Post by SilverRed »

I had a really good go at reading this and trying to understand it but I feel I don't understand a whole lot more about image decoding/encoding than before I started. Like I get the core concepts of key frames, motion vectors and such on a high level but if you asked me to actually create a decoder I wouldn't have a clue where to start.

I feel like I would need a full hour long video on each paragraph of this post to really understand it.

Re: Decoding AVIF: Deep dive with cats and imgproxy

#15

Post by keithwinstein »

dylan604 wrote: post #2

"Commonly, three numbers are used to specify downsampling: the first is always 4, don’t ask me why" Wow. Someone going to this much detail on explaining how a video/image codec works, and cannot bother learning what the numbers of chroma subsampling mean? The first number represents the luminance.[0] Even if they know the first number represents luminance, the "don't ask me why" is just horrible on its own. The detai…

Poynton has a pretty plausible-sounding explanation here (https://poynton.ca/PDFs/Chroma_subsampling_notation.pdf):

"The commonly used leading digit of 4 is a historical reference to a sample rate roughly four times the NTSC or PAL color subcarrier frequency; the notation originated when subcarrier-locked sampling was under discussion for component video. Upon the adoption of component video sampling at 13.5 MHz, the first digit came to specify luma sample rate relative to 3 3⁄8 MHz. HDTV was once supposed to be described as 22:11:11! Since then, the leading digit has – thank-fully – come to be relative to the sample rate in use. Until recently, the initial digit was always 4, since all chroma ratios have been powers of two – 4, 2, or 1. However, 3:1:1 subsampling has been commercialized in an HDTV production system (Sony’s HDCAM), so 3 may now appear as the leading digit. By convention, a leading digit of 2 is never used."

And here is lots of detailed history: https://tech.ebu.ch/docs/techreview/trev_304-rec601_wood.pdf , including a lot of debate in the late 70's about "three-times sub-carrier (3fsc) versus four-times sub-carrier (4fsc) sampling." The victory for team "4" is, I think, why that's the leading digit, even though they ended up compromising on not-quite-4 in the end.

Re: Decoding AVIF: Deep dive with cats and imgproxy

#16

Post by keithwinstein »

zinekeller wrote: post #9
dylan604 wrote: post #2

"Commonly, three numbers are used to specify downsampling: the first is always 4, don’t ask me why" Wow. Someone going to this much detail on explaining how a video/image codec works, and cannot bother learning what the numbers of chroma subsampling mean? The first number represents the luminance.[0] Even if they know the first number represents luminance, the "don't ask me why" is just horrible on its own. The detai…

Edit: While the reason for 2 vertical lines (interlacing) is correct, it seems that the 4 horizontal lines is from a compromise between NTSC and 525-line systems' frequency (explained in length here: https://news.ycombinator.com/item?id=28203942 , thank you keithwinstein), standardised as Rec. 601. One of the proposals is 3:2:x, but it was both worse analogue-speaking and harder digitally-speaking. 3:1:x was used for…

I don't think this is quite right. My understanding is that 4:2:2 chroma subsampling for Rec. 601 video (and the choice of sampling at 4x the chroma sub-carrier) came around in the days of 525/59.94 vs. 625/50 systems, before even the Japanese analog HDTV systems -- see my link above (https://tech.ebu.ch/docs/techreview/trev_304-rec601_wood.pdf).

Re: Decoding AVIF: Deep dive with cats and imgproxy

#17

Post by zinekeller »

keithwinstein wrote: post #16
zinekeller wrote: post #9

Earlier quoted context omitted.

Edit: While the reason for 2 vertical lines (interlacing) is correct, it seems that the 4 horizontal lines is from a compromise between NTSC and 525-line systems' frequency (explained in length here: https://news.ycombinator.com/item?id=28203942 , thank you keithwinstein), standardised as Rec. 601. One of the proposals is 3:2:x, but it was both worse analogue-speaking and harder digitally-speaking. 3:1:x was used for…

I don't think this is quite right. My understanding is that 4:2:2 chroma subsampling for Rec. 601 video (and the choice of sampling at 4x the chroma sub-carrier) came around in the days of 525/59.94 vs. 625/50 systems, before even the Japanese analog HDTV systems -- see my link above ( https://tech.ebu.ch/docs/techreview/trev_304-rec601_wood.pdf ).

Thank you for pointing this - I've checked engineering notes from that time and it was indeed mentioned (it was even mentioned in proto-D-MAC from 1981), I'll correct the post. I wonder though how I missed that, worse forgetting Rec. 601!

Re: Decoding AVIF: Deep dive with cats and imgproxy

#18

Post by magicalhippo »

SilverRed wrote: post #14

I had a really good go at reading this and trying to understand it but I feel I don't understand a whole lot more about image decoding/encoding than before I started. Like I get the core concepts of key frames, motion vectors and such on a high level but if you asked me to actually create a decoder I wouldn't have a clue where to start. I feel like I would need a full hour long video on each paragraph of this post to…

We made a mpeg-ish encoder and decoder at uni, and it was surprisingly simple.

Keyframes were just jpegs. Then for intraframes, you first found the motion vectors for each 8x8 block, then generated the predicted intraframe from the keyframe and motion vector. Then you subtracted the prediction off the keyframe. The resulting "prediction error image" was then simply jpeg encoded as the intraframe, and appended to the output after the motion vectors.

Decoding was reverse, reconstruct the predicted intraframe from the previous keyframe and motion vectors, and add back the prediction error image.

Might be glossing over something as it was over a decade ago but should be about the gist of it.

We played with different algorithms for finding motion vectors and such, including accelerating it with GPGPU.

Really fun project, and assuming you use an existing jpeg library, not at all big or difficult.

Re: Decoding AVIF: Deep dive with cats and imgproxy

#19

Post by newpavlov »

HALtheWise wrote: post #11
dylan604 wrote: post #2

"Commonly, three numbers are used to specify downsampling: the first is always 4, don’t ask me why" Wow. Someone going to this much detail on explaining how a video/image codec works, and cannot bother learning what the numbers of chroma subsampling mean? The first number represents the luminance.[0] Even if they know the first number represents luminance, the "don't ask me why" is just horrible on its own. The detai…

One thing I never understood is why _downsampling_ is the most efficient way to compress the data about chroma into fewer bits while maximizing perceptual accuracy. It really seems like for any given target bitrate for the chroma data, there should always be a more efficient compression scheme available than simply throwing out 3/4 of the pixels and running compression algorithms on the rest. Surely modern compressio…

It makes sense not only from biological point of view as noted in the article, but also from technological as well. Almost all color cameras use Color Filter Arrays [1], meaning that for WxH resolution you don't get WxHx3 pixel values as you would expect from RGB images which you usually consume, but only WxH (i.e. 2/3rds of RGB image data is generated, not measured). With 4:4:4 sampling you have 12 values per 2x2 block, even though only 4 values have been measured by camera for it. Meanwhile with 4:2:0 sub-sampling you have 6 values, which is still bigger than 4, but quite convenient for processing in Y-based color spaces.

[1]: https://en.wikipedia.org/wiki/Color_filter_array

Post reply on HN