New algorithm unlocks high-resolution insights for computer vision

They reproject the input images and run the low-res network multiple times. Then they use an approach similar to NeRF to merge the knowledge from those reprojected images into a super-resolution result.

So in a way, this is quite similar to how modern Pixel phones can take a burst of frames and merge them into a final image that has a higher resolution than the sensor. Except that they run useful AI processing in between and then do the super-resolution merge on the results.

Re: New algorithm unlocks high-resolution insights for computer vision

#5

Post by skybrian » Tue, Mar 19, 2024, 10:34 PM UTC

It's not that clear why they are downsampling and then upsampling again. Why not do all the work at the original resolution?

Apparently, the issue is that some vision algorithms only output a low-res representation and that needs to be upsampled to match the original?

Re: New algorithm unlocks high-resolution insights for computer vision

#6

Post by famouswaffles » Tue, Mar 19, 2024, 10:43 PM UTC

skybrian wrote: post #5

It's not that clear why they are downsampling and then upsampling again. Why not do all the work at the original resolution? Apparently, the issue is that some vision algorithms only output a low-res representation and that needs to be upsampled to match the original?

>It's not that clear why they are downsampling and then upsampling again. Why not do all the work at the original resolution?

For NNs, This is pretty much a compute efficiency thing. Working on the original resolution directly is more compute intensive.

Re: New algorithm unlocks high-resolution insights for computer vision

#7

Post by 3abiton » Tue, Mar 19, 2024, 11:22 PM UTC

fxtentacle wrote: post #4

What an amazing idea :) They reproject the input images and run the low-res network multiple times. Then they use an approach similar to NeRF to merge the knowledge from those reprojected images into a super-resolution result. So in a way, this is quite similar to how modern Pixel phones can take a burst of frames and merge them into a final image that has a higher resolution than the sensor. Except that they run use…

Very interesting, I am curious how do people reach that train of thought to a successful idea. So many great algorithms based on small twists.

Re: New algorithm unlocks high-resolution insights for computer vision

#8

Post by refulgentis » Tue, Mar 19, 2024, 11:22 PM UTC

skybrian wrote: post #5

It's not that clear why they are downsampling and then upsampling again. Why not do all the work at the original resolution? Apparently, the issue is that some vision algorithms only output a low-res representation and that needs to be upsampled to match the original?

Correct, s/some/vast majority of. Ex. major video conference software ML blur algos run at like 100x100 - the weird edge is much more about resolution of input/output than ML.

Re: New algorithm unlocks high-resolution insights for computer vision

#9

Post by kreelman » Tue, Mar 19, 2024, 11:39 PM UTC

This looks like it could be useful. Remote sensing uses feature extraction tools. Being able to upsample again would make the data a lot easier to view and interpret. Nice work.

Re: New algorithm unlocks high-resolution insights for computer vision

#10

Post by kreelman » Tue, Mar 19, 2024, 11:45 PM UTC

3abiton wrote: post #7

fxtentacle wrote: post #4

What an amazing idea :) They reproject the input images and run the low-res network multiple times. Then they use an approach similar to NeRF to merge the knowledge from those reprojected images into a super-resolution result. So in a way, this is quite similar to how modern Pixel phones can take a burst of frames and merge them into a final image that has a higher resolution than the sensor. Except that they run use…

Very interesting, I am curious how do people reach that train of thought to a successful idea. So many great algorithms based on small twists.

It is interesting indeed. One wonders if the researchers of this particular bit of work made it mandatory to go for walks at lunch and think about how their vision chunked/filtered the information it was receiving. Interesting that they "perturb" the image to get some noise involved. I'll need to read it over again.