Live data from Hacker News

Nvidia Uses AI to Slash Bandwidth on Video Calls

petapixel.com

131–140 of 197 posts

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#131

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

That's how your brain works as well.

Adding a secondary, uncontrolled layer of perception is definitely not the same thing, but in general, what you "see" is barely "real".

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#132

> they have managed to reduce the required bandwidth for a video call by an order of magnitude. In one example, the required data rate fell from 97.28 KB/frame to a measly 0.1165 KB/frame – a reduction to 0.1% of required bandwidth. A nitpick, perhaps, but isn't that three orders of magnitude? We've already seen people use outlandish backgrounds in calls, now it's going to be possible to design similar outlandish vie…

This reminds me of a sci-fi novel I read in the nineties. The premise had something to do with actors who took on roles in virtual reality where their bodies are fit with sense-points. They're cast in live-action role-plays with wealthy remote clients. They're basically deep-fakes in VR.

Neal Stephenson's Diamond Age had an element like that. It was even possible that the actors ("ractors") didn't necessarily know what they were acting in, just the general parameters and the next set of lines and actions necessary to continue the performance.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#133
I can steal all your video data of 'you' and call someone else back, as you?

I could even get an accomplice to do it while I'm talking to you. They would have your today clothes on and you'd be tied up talking to me.

I'm dubious on the tech being as good as they say now. But it's getting exciting.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#134
Great concept. With higher quality input, keyframe, point tracking, and ML model the output should improve significantly with similar bandwidth improvement. I think this could also smooth between dropped frames and offer higher bandwidth for the audio feed.

The issues are social. I would hope that the receiver is the one able to choose between original or AI stream, as I can understand some people being uncomfortable with the artifacts, gaze, expressions, and other edge cases. But when the quality is higher I could see a lot of people preferring this option as a default.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#135
post #97

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

your brain does lots of preprocessing to create a stable view of "reality", glossing over blind spots, retinal blood flow, etc. in a way we are always hallucinating, and it seems natural that computers should learn what we are actually paying attention to in video, and focus on that

Yes! E.g. AAA games render the entire 4k screen with the highest graphics, but only a small spot hits the part of retina that can perceive jagged edges or lighting details.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#136

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

The raw sensor data for a digital camera looks something like this https://commons.wikimedia.org/wiki/File:Normal_and_Bayer_Fil...

In addition to the colour grading and interpolation, lens defects might be 'fixed' before you ever get data on disk.

ML is just really good interpolation.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#137
post #110

Earlier quoted context omitted.

Video is not real either. The perspective is different and the colors are wrong. It is even less real after it went through lossy compression. The whole point of a lossy compression is to remove details that you perceive as unimportant. For example, leaves on a tree may look like a greeny mess, but that's fine, from afar, you don't make a difference. Using neural networks for compression is far from being a new conce…

In some ways, of course, I agree, but as some other commenters have pointed out, this is essentially a "deepfake" of yourself...in principle there's no restriction on what one could use as their "avatar." As much was demonstrated in the "Different Output Styles" slide. So I'd have a hard time saying this is not dramatically less "real" than some lossy compression technique. Is there some way to formalize "realness?"…

Maybe simply using an objective video quality measurement against the original picture would do the trick. Something like PSNR or SSIM. A "deepfake" is likely to score low on such a score that doesn't depend on high level visual perception.

Also know that you can "deepfake" yourself using a traditional video encoder, just change the keyframe to someone else's face. Of course, it will look broken and totally unconvincing but because of motion compensation, you can sort of map the movement of your face on someone else's face.

The technique in the paper simply has way better motion compensation, so good that it still works if you change the keyframe. Traditional video compression algorithms don't work like that because they are not just for talking faces and can't use such advanced techniques for performance and ease of implementation reasons.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#138
post #62

Earlier quoted context omitted.

You just need to connect GPT3, and the dialogue is taken care of. Lyrebird’s API will take care of the speech synthesis. Viola! My deep fake can stand in at meetings now while I code.

Imagine if everyone did that

I posit that no productivity would be lost. Just have a summarizing AI email you the outcomes.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#139
post #110

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

Video is not real either. The perspective is different and the colors are wrong. It is even less real after it went through lossy compression. The whole point of a lossy compression is to remove details that you perceive as unimportant. For example, leaves on a tree may look like a greeny mess, but that's fine, from afar, you don't make a difference. Using neural networks for compression is far from being a new conce…

The distinction that makes ai compression creepy is that it can use prior knowledge of what a face is.

Traditional compression has no high level knowledge of what a video call looks like, the algorithms are about patterns in how pixels change in time and space, so the artifacts from the algorithm are pixel-based effects (blockiness, blurriness, etc).

A neural net that has been trained on a million faces and is setup to draw a face may draw a perfectly clear image of a face on low bandwidth...but when it doesn't have bandwidth to be accurate, it doesn't blur things, it fills in details learned from looking at strangers who aren't on the call.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#140
post #62

Earlier quoted context omitted.

Imagine if everyone did that

I posit that no productivity would be lost. Just have a summarizing AI email you the outcomes.

Hello, isoprophlex, I'm your Assistant AI

Today's meeting was an hour-and-a-half spent on bike-shedding the position--and color--of the "logout" button on our product page.

15 minutes were spent debating the resident usability expert who suggested white text on a dark blue background would be more readable for people with low vision. The department manager insisted on retaining pastel blue as it is his favorite color.

45 minutes were spent arguing over whether "logout" or "log out" comprises proper semantics. Our linguistic expert was unfortunately not able to attend as she was sent on a business trip earlier in the week.

The last 30 minutes were focused on team-building exercises as Bob doodled on his tablet and Susan smiled politely at the speaker whilst screaming internally as she had 3 more meetings to attend before her department could move forward.

Post reply on HN