Earlier quoted context omitted.
You mean _invents_ details.
You mean _infers_ details.
Or, extracts from its digital rectum?
VideoGigaGAN: Towards detail-rich video super-resolution
41–50 of 243 posts
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#42Earlier quoted context omitted.
That's absurd. I think anybody is aware that it is far superior to e.g. compress in the frequency domain than to down sample your image. If you don't believe me just compare a JPEG compressed image with the same image of the same size compressed with down sampling. You will notice a literal night and day difference. Down sampling is a bad way to do compression. It makes no sense to do NN reconstruction on that if you…
Are you saying that when Netflix streams a 480p version of a 4k movie to my TV they do not perform downsampling?
Yes. Down sampling makes only sense if you store per pixel data, which is obviously a dumb idea. You get a stream for 480p which contains frames which were compressed from the source files, or the 4k version. At some point there might have been down sampling involved, but you never actually get any of that data, you get the compressed version of those.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#43Wonder how long until Hollywood CGI shops have these types of models running as part of their post-production pipeline. Big blockbusters often release with ridiculously broken CGI due to crunch (Black Panther's third act was notorious for looking like a retro video-game), adding some extra generative polish in those cases is a no-brainer.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#44Earlier quoted context omitted.
There's lots of videos where there isn't high res data available
Totally irrelevant to the discussion, which is explicitly about streaming services delivering in lower resolutions than they have available.
Streaming services already deliver in lower resolution than they available based on network conditions. Good upscaling would let you save on bandwidth and deliver content easier to people in poor network conditions. The tradeoff would be that details in the image wouldn't be exactly the same as the original - but, presumably, nobody would notice this so it would be fine.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#45Video quality seems really good, but limitations are quite restrictive "Our model encounters challenges when processing extremely long videos (e.g. 200 frames or more)". I'd say most videos in practice are longer than 200 frames, so lot more research is still needed.
Fascinating how researchers put out amazing work and then claim that videos consisting of more than 200 frames are "extremely long". Would it kill them to say that the method works best on short videos/scenes?
Tale as old as time, in graphics papers it's "our technique achieves realtime speeds" and then 8 pages down they clarify that they mean 30fps at 640x480 on an RTX 4090.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#46This is amazing and all but at what point do we reach the point of there is no more “real” data to infer from low resolution? In other words there are all sorts of information theory research on the amount of unique entropy on a given medium and even with compression there is a limit. How does that limit relate to work like this? Is there a point at which it can say we know it’s inventing things beyond x scaling constant because of information theory research?
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#47Have we reached peak image sensor size. Would it still make sense to shoot in fullframe when you can just upscale.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#48Earlier quoted context omitted.
I agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p version and let the TV create the 4K detail.
So, DLAA for video instead of games? https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing
Not really, DLAA and the current incarnation of DLSS are temporal techniques, meaning all of the detail they add is pulled from past frames. That's an approach which only really makes sense in games where you can jitter the camera to continuously generate samples at different subpixel offsets with each frame.
The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air rather than from previous frames, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#49Earlier quoted context omitted.
>Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity Because you can just not downscale them and compress them in the frequency domain and encode them in 200Kbps? This is pretty obvious, seriously do you not understand what JPEG does? And why it doesn't do down sampling? Do you seriously believe downscalin…
4:2:0 which is used in all common video codecs is down scaling the color data.
Scaling color data is a different technique than down sampling. Again, all I am saying is that for a very good reason you do not stream pixel data or compress movies by storing data that was down sampled.
Re: VideoGigaGAN: Towards detail-rich video super-resolution
#50Earlier quoted context omitted.
There is no use case, because it is a stupid idea. Downscaling then reconstructing is a stupid idea for exactly the same reasons why downscaling for compression is a bad idea. The issue isn't NN reconstruction, but that you are reconstructing the wrong data.
if the nn is part of the codec, you can choose to only downscale the regions that get reconstructed correctly.
Why would you not let the NN work on the compressed data? That is actually where the information is.