Arbitrary-Scale Super-Resolution with Neural Heat Fields
therasr.github.io
Arbitrary-Scale Super-Resolution with Neural Heat Fields
1–10 of 59 posts
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#2The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space, why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space?
Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spend more of their compute budget outputting detail that's relevant to human vision. Color spaces that split brightness from chroma would allow increased contrast detail and lower color detail. This is basically what JPG does.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#3The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…
> why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space
I've been wondering exactly this for a while, if somebody more knowledgeable knows why we're not doing that I'd be happy to hear it.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#4The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…
You may already know this, but image generators like Stable Diffusion and Flux already do this in the form of “latent diffusion”.
Rather than operate on pixel space directly, they learn to operate on images that have been encoded by a VAE (latents). To generate an image with them, you run the reverse diffusion (actually flow in the case of flux) process they’ve learned and then decode the result using the VAE.
These VAE encoded latent images are 8x smaller in width/height and have 4 channels in the case of Stable Diffusion and 16 in the case of Flux.
I do think it would be more useful if it worked more like you said, though - if the channels weren’t encoded arbitrarily but some of them had pretty clear, useful human meaning like lightness, it would be another hook to control image generation.
To some extent, you can control the existing VAE channels, but it is pretty finicky.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#5The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…
We do, see eg LPIPS loss
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#6Where are the ground truth images?
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#7Where are the ground truth images?
Click through to the actual paper and they are in the last column labeled “GT”.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#8Seems like a nice result but wouldn’t have hurt for them to give a few performance benchmarks. I understand that the point of the paper was a quality improvement, but it’s always nice to reference a baseline for practicality.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#9The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…
You may already know this, but image generators like Stable Diffusion and Flux already do this in the form of “latent diffusion”. Rather than operate on pixel space directly, they learn to operate on images that have been encoded by a VAE (latents). To generate an image with them, you run the reverse diffusion (actually flow in the case of flux) process they’ve learned and then decode the result using the VAE. These…
If there's one thing that neural networks have shown, it's that they are much better at picking up encoding patterns for realistic tasks than humans. There are so many aspects that could be used in dimensional reduction tasks that it seems pretty wild that we've come this far with human-designed patterns. From a top down engineering perspective, it might seem like a disadvantage to have algorithms that are not tailored to particular cases. But when you want things like general purpose image generation, it's simply much more economical to let ML figure out which dimensions to focus on. Because humans would spend years coming up with the details of certain formats and still not cover half the cases.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#10I'd like to see the results in something like Wing Commander Privateer.