@0x12A what’s the difference between this version and v1 of the paper from November 2023?
Arbitrary-Scale Super-Resolution with Neural Heat Fields
41–50 of 59 posts
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#42The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…
There is definitely work out there that deals directly in dct blocks from jpeg:
https://arxiv.org/abs/1907.11503
https://arxiv.org/abs/2308.09110
With generative ai they tend to have a learned compressed representation instead (VAE)
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#43I'd like to see the results in something like Wing Commander Privateer.
It does not work with pixel art very well.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#44Could such method be adapted to audio? For instance to upscale 8-bit samples to 16-bit in Amiga mods?
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#45Earlier quoted context omitted.
Not disagreeing, but the number of parameters are listed in the single digit millions size (which surprised me). So, I would expect this to be very fast on modern hardware.
Very fast is a bit vague in a space where you might have a millisecond per frame.
This seems more in line with potential real-time applications, though it is still a lot slower than DLSS: https://dl.acm.org/doi/10.1145/3641519.3657439
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#46Why do those algorithm not include prompting to guide the scaling ?
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#47The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…
Lossy image compression has mostly targeted an entirely different performance envelope.
E.g. in the image you can see a diagonal bands basis function. Image codecs don't generally have those-- not because they wouldn't be useful but because codec developers favor separable transforms that have fast factorizations for significant performance improvements.
I don't think we know and can really make good comparisons between traditional tools and ML powered compression because of this. We just don't have decades of efforts where the engineers were allowed a million multiples and a thousand memory accesses per pixel.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#48Very good work! Sadly this model really does not like nosy images that have codec compression artifacts, at least with my few test images.
I wonder if there is a de-artifacting model out there.
I think the company named Topaz had a photoshop plugin to remove "jpeg artifacts" - I don't know if they are using a neural model for it though.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#49Was anyone else expecting an infinitely zoomable pictures from that title? I am disappoint
You were imagining something where you give it one grey pixel, then zoom in infinitely and read the Magna Carta? Where did you imagine it would get the information from?
Wait but that’s the point. It doesn’t have the information, it makes a best guess to match its training.
Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields
#50Instead of training on vast amounts of arbitrary data that may lead to hallucinations, wouldn't it be better to train on high-resolution images of the specific subject we want to upscale? For example, using high-resolution modern photos of a building to enhance an old photo of the same building, or using a family album of a person to upscale an old image of that person. Does such an approach exist?
Author here -- Generally in single image super-resolution, we want to learn a prior over natural high-resolution images, and for that a large and diverse training set is beneficial. Your suggestion sounds interesting, though it's more reminiscent of multi image super-resolution, where additional images contribute additional information, that has to be registered appropriately. That said, our approach is actually trai…
It feels like it's multishot nl-means, then immedeately those pre-trained "AI upscale" things like Topaz with nothing in between. Like, if I have 500 shots from a single session and I would like to pile the data together to remove noise and increase detail, preferably starting from the raw data, then - nothing? Only guys doing something like that are astrophotographers, but their tools are .. specific.
But for "normal" photography, it is either pre-trained ML, pulling external data in, or something "dumb" like anisotrophic blurring.