If you want a similar read on DCT-based encoding, I highly recommend this article which lays it out in an extremely digestible format:
Fighting JPEG color banding
41–50 of 53 posts
Re: Fighting JPEG color banding
#42> Ok, but where does this table come from when we need to save a file? It would be a big complication if you had to construct, and transmit 64 independent numbers as a parameter. Instead, most encoders provide a simple interface to set all 64 values simultaneously. This is the well known “quality,” which value could be from 0 to 100. So, we just provide the encoder desired quality and it scales some “base” quantization table. The higher quality, the lower values in quantization table.
I never really thought about how that "quality" slider worked (besides making the compression lossier), but it makes perfect sense now! It always amazes me how much I take for granted.
I always treat compression like a black box: "-crf 23" for H264, PNG and FLAC are nice but MP3 320s and 90+ "quality" JPEGs are good compromises, etc. And that's just for the stuff I deal with, there's no telling how much lossy compression goes on behind the scenes on my own computers, let alone all the stuff served up over the internet. There's so much lossy compression in the world, from EP speed on VHS tapes to websites reencoding uploaded images to every online video ever, it's crazy to think about.
Re: Fighting JPEG color banding
#43I love this and would have dearly needed it like 5 years ago. Now, it is still a very interesting read. But given what we have already seen from Nvidia on video compression [0], I think within the next few years, we will move everything to machine-learning-'compressed' images (aka transmitting a super-low-res seed image and some additional ASCII and having an ML model reconstruct and upscale it at the client side). […
honestly, this scares this shit out of me. lossy compression is one thing, but to just say that an ML model suggests making pixels like this vs a mathematical formula is totally different things.
What if it's an ML model suggesting parameters for jpeg? It could still hallucinate to some degree, but it's also more limited.
Re: Fighting JPEG color banding
#44Earlier quoted context omitted.
honestly, this scares this shit out of me. lossy compression is one thing, but to just say that an ML model suggests making pixels like this vs a mathematical formula is totally different things.
What if it's an ML model suggesting parameters for jpeg? It could still hallucinate to some degree, but it's also more limited.
Image -> mathematical forumla to toss data -> reverse formula -> slightly altered image
vs
Image -> mathematical formula to toss data -> ML to recreate what it thinks is supposed to be there -> made up image based on "training" data not even from original image
that's my problem
Re: Fighting JPEG color banding
#45Earlier quoted context omitted.
What if it's an ML model suggesting parameters for jpeg? It could still hallucinate to some degree, but it's also more limited.
Image -> mathematical forumla to toss data -> reverse formula -> slightly altered image vs Image -> mathematical formula to toss data -> ML to recreate what it thinks is supposed to be there -> made up image based on "training" data not even from original image that's my problem
ML based image compression don't generally let the net make up data, they use a net as a prior to reduce the entropy of the data that's there.
Re: Fighting JPEG color banding
#46Earlier quoted context omitted.
What if it's an ML model suggesting parameters for jpeg? It could still hallucinate to some degree, but it's also more limited.
Image -> mathematical forumla to toss data -> reverse formula -> slightly altered image vs Image -> mathematical formula to toss data -> ML to recreate what it thinks is supposed to be there -> made up image based on "training" data not even from original image that's my problem
What I'm saying is I'm not sure how much ML can imagine just by changing coefficient precision.
Re: Fighting JPEG color banding
#47Earlier quoted context omitted.
Turns out there is a project which does that: https://github.com/google/guetzli
Guetzli is really hamstrung by its resource usage. When it first hit the news I tried it out, and compressing a full quality JPEG from my phone could take 20-30 minutes on an i7.
It depends on the context. If I'm converting a whole library of photos I wouldn't use it. But I've got a big hero area JPEG that's loaded as one of the first resources - I'm happy to run this tool in the background for a day to make it 20% smaller.
Re: Fighting JPEG color banding
#48just use PNG. JPEG sucks.
PNG sucks for some uses too. There is no universally perfect image format for all possible use cases.
WebP & JPEG XL compress losslessly better than PNG and lossy much better than JPEG. Perhaps not perfect either, but we finally do have formats that can do both — and better than either before.
Re: Fighting JPEG color banding
#49There's also webp and heif and png and svg, and I believe all the existing formats already solve the image compression problem. The difference of 18kb vs 22kb from hours of microoptimisations is frankly irrelevant given the rate of networks getting faster.
The simple question: if all the existing formats already solve the image compression problem, why a new image formats (WebP, HEIC, AVIF, JPEG XL etc) appears?
As computers generally get faster, more expensive encoding can be considered.
I'm using different tooling on an AMD 3900x for these conversions, so take these numbers with a grain of salt.
$ gm identify test.tiff
test.tiff TIFF 6240x4160+0+0 DirectClass 8-bit 1.2Mi 0.000u 0m:0.000001s
$ time gm convert test.tiff test.jpg
real 0m0.282s
user 0m0.193s
sys 0m0.089s
$ time heif-enc test.jpg -o test.heif
real 0m1.901s
user 0m22.960s
sys 0m0.180s
In my prior tests AV1 is 2-5x slower than HEIF, and JPEG-XL is ~10x slower.
Re: Fighting JPEG color banding
#50Earlier quoted context omitted.
What if it's an ML model suggesting parameters for jpeg? It could still hallucinate to some degree, but it's also more limited.
Image -> mathematical forumla to toss data -> reverse formula -> slightly altered image vs Image -> mathematical formula to toss data -> ML to recreate what it thinks is supposed to be there -> made up image based on "training" data not even from original image that's my problem
When you think "ML to recreate what it thinks is supposed to be there" you probably automatically go to DeepDream or https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...
but the end result doesnt have to be direct output of ML hallucination. AI encodes probability distribution, you can treat it as motion compensation in video codecs - what comes next is a convolution by encoded error between predicted outcome and ground truth.