Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arbitrary-Scale Super-Resolution with Neural Heat Fields

therasr.github.io

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields

#21

Post by 0x12A »

adhoc32 wrote: post #14

Instead of training on vast amounts of arbitrary data that may lead to hallucinations, wouldn't it be better to train on high-resolution images of the specific subject we want to upscale? For example, using high-resolution modern photos of a building to enhance an old photo of the same building, or using a family album of a person to upscale an old image of that person. Does such an approach exist?

Author here -- Generally in single image super-resolution, we want to learn a prior over natural high-resolution images, and for that a large and diverse training set is beneficial. Your suggestion sounds interesting, though it's more reminiscent of multi image super-resolution, where additional images contribute additional information, that has to be registered appropriately.

That said, our approach is actually trained on a (by modern standards) rather small dataset, consisting only of 800 images. :)

Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields

#23

Post by pizza »

crazygringo wrote: post #12
jiggawatts wrote: post #2

The learned frequency banks reminded me of a notion I had: Instead of learning upscaling or image generation in pixel space , why not reuse the decades of effort that has gone into lossy image compression by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) with a perceptually uniform color space like UCS. This would allow models to be optimised so that they spen…

> by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) I've never understood the DCT to be psychovisually optimal at all. At lower bitrates, it degrades into ringing and blockiness that don't match a "simplified perception" at all. The frequency domain models our auditory space well, because our ears literally process frequencies. Bringing that over to the visual…

JPEG is good for when you want a picture to look reasonably good while throwing away ~90-95% of the data. In fact, there's a relatively new JPEG variant that lets you get even better psychovisual fidelity for the same compression level by just doing JPEG in the XYB color space, xybjpeg. JPEG is also a very simple algorithm, when compared to the ones that'd be noticeably better near 99% compression.

To beat blockiness/banding across very gradually varying color gradients (think eg the gradient of a blue sky), JPEG XL has to whip out a lot of tricks, like handling sub-LF DCT coefficients between blocks, heterogeneous block sizes, deblocking filters for smoothing, and heterogeneous quantization maps.

BTW, one of the ways different camera manufacturers aimed to position themselves as having cameras that generated the best pictures was by using custom proprietary quantization tables to optimize for psychovisual quality.

Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields

#25

Post by crazygringo »

pizza wrote: post #23
crazygringo wrote: post #12

Earlier quoted context omitted.

> by generating output in a psychovisually optimal space? Perhaps frequency space (discrete cosine transform) I've never understood the DCT to be psychovisually optimal at all. At lower bitrates, it degrades into ringing and blockiness that don't match a "simplified perception" at all. The frequency domain models our auditory space well, because our ears literally process frequencies. Bringing that over to the visual…

JPEG is good for when you want a picture to look reasonably good while throwing away ~90-95% of the data. In fact, there's a relatively new JPEG variant that lets you get even better psychovisual fidelity for the same compression level by just doing JPEG in the XYB color space, xybjpeg. JPEG is also a very simple algorithm, when compared to the ones that'd be noticeably better near 99% compression. To beat blockiness…

No disagreements.

I do suspect that at some point we will make a major compression breakthrough that is based on something more "psychovisual". Not Gaussian splatting, but something more akin to that -- something that directly understands geometric areas of gradating colors as primitive objects, textures as primitives, and motion as assigned to those rather than to pixels.

On the other hand, it may very well be a form of AI-based compression that does this, rather than us explicitly designing it.

Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields

#26

Post by crazygringo »

adhoc32 wrote: post #14

Instead of training on vast amounts of arbitrary data that may lead to hallucinations, wouldn't it be better to train on high-resolution images of the specific subject we want to upscale? For example, using high-resolution modern photos of a building to enhance an old photo of the same building, or using a family album of a person to upscale an old image of that person. Does such an approach exist?

That is how Hollywood currently de-ages famous actors, by training on their photos and stills from when they were around the desired age.

But it's extremely time-consuming and currently expensive.

Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields

#27

Post by 0x12A »

mrybczyn wrote: post #24

hrm. on nature portrait photography 600x600 upscale, it has a LOT of artifacts. Perhaps too far out of distribution? That said, your examples are promising, and thank you for posting a HF space to try it out!

Hi, author here :) It shouldn’t be OOD, unless its too noisy maybe? And what scaling factor did you use? Single image SR is a highly ill-posed problem, so at higher upscaling factors it just becomes really difficult…

Re: Arbitrary-Scale Super-Resolution with Neural Heat Fields

#30

Post by KeplerBoy »

vessenes wrote: post #13
WhitneyLand wrote: post #8

Seems like a nice result but wouldn’t have hurt for them to give a few performance benchmarks. I understand that the point of the paper was a quality improvement, but it’s always nice to reference a baseline for practicality.

Not disagreeing, but the number of parameters are listed in the single digit millions size (which surprised me). So, I would expect this to be very fast on modern hardware.

Very fast is a bit vague in a space where you might have a millisecond per frame.
Post reply on HN