Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Uses AI to Slash Bandwidth on Video Calls

petapixel.com

Post reply on HN
Print view View original

151–160 of 197 posts

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#151

Post by alkonaut »

varispeed wrote: post #95

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally sends both things, then the receiver should be able to always piece together a 100% reproduction of the actual video feed on my end. The transmitter can always predict exactly what the receiver will decode, so the correct amount of data to send is the amount of data that makes the receiver see what I want. If that means I send 0.1mbps key point data and 1mbps diff pixels then that’s what I want to send.

Is that how this works? Because it should be how it works...

Not sure if this is a viable way of compressing the video stream or if a transmission of the “diff” would be too costly. If this method would give 1/2 the original bandwidth I’d think it’s more impressive than cutting 3 orders of magnitude via “avatars”.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#152

Post by deeringc »

edoloughlin wrote: post #141

* instead of sending a stream of pixel-packed images, it sends specific reference points on the image around the eyes, nose, and mouth* So they're trading bandwidth for CPU load at either end. I wonder what the tradeoff is in terms of energy? Would this result in higher CO2 emissions?

> So they're trading bandwidth for CPU load at either end

Given it's Nvidia I would imagine it's more likely going to be GPU load rather than CPU load. Don't underestimate the current computational overhead of existing lossy video compression.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#153

Post by cordite »

polytely wrote: post #102
OrderlyTiamat wrote: post #12

> they have managed to reduce the required bandwidth for a video call by an order of magnitude. In one example, the required data rate fell from 97.28 KB/frame to a measly 0.1165 KB/frame – a reduction to 0.1% of required bandwidth. A nitpick, perhaps, but isn't that three orders of magnitude? We've already seen people use outlandish backgrounds in calls, now it's going to be possible to design similar outlandish vie…

I think you are right on the money with your thoughts on virtual avatars. I've already noticed this phenomenon cropping up in some niches. 1. the phenomenon of VTubers https://en.m.wikipedia.org/wiki/Virtual_YouTuber 2. in the virtual animal crossing late night show, Animal Talking, the presenter's (Gary Whitta) avatar doesn't really resemble how the presenter looks in real life https://en.m.wikipedia.org/wiki/Animal…

This sounds a lot like the movie Surrogates, where at one point the protagonist notices the badge of a android surrogate is completely different from the human behind the surrogate (as printed on the badge).

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#154

Post by Fluid_Mechanics »

pretendscholar wrote: post #96
TuringNYC wrote: post #48

Earlier quoted context omitted.

You just need to connect GPT3, and the dialogue is taken care of. Lyrebird’s API will take care of the speech synthesis. Viola! My deep fake can stand in at meetings now while I code.

GPT will also code for you.

Great, I can spend more time gratifying my limbic system while accumulating resources to survive my impending obsolescence.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#155

Post by kevincox »

alkonaut wrote: post #151
varispeed wrote: post #95

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally sends both things, then the receiver should be able to always piece together a 100% reproduction of the actual video feed on my end. The transmitter can always predict exactly what the receiver will decode, so the correct amount of data to send is the amount o…

I suspect that this isn't feasible. Otherwise lossy video codecs wouldn't exist (or would be very niche). Because if you sent a perfect diff you would essentially have lossless compression again.

So either your idea will revolutionize video compression or the "diff" would bring you back to the ballpark of lossless codecs.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#156

Post by iandanforth »

GuB-42 wrote: post #110
varispeed wrote: post #95

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

Video is not real either. The perspective is different and the colors are wrong. It is even less real after it went through lossy compression. The whole point of a lossy compression is to remove details that you perceive as unimportant. For example, leaves on a tree may look like a greeny mess, but that's fine, from afar, you don't make a difference. Using neural networks for compression is far from being a new conce…

"the result is not more or less real than any other technique"

I disagree. If real / not real is binary, sure, it's not real. But if we allow "real" to be a range of values, it is less real.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#157

Post by motoboi »

dhdhhdd wrote: post #149

At what resolution? And also, does the output actually resembles the original image? Examples with background other than uniform? Would be nice if they provided more than just screenshots It's not uncommon to see video calls at 100kbs-150kbps, which is ~10KB/s, and this is for 7fps or so, including audio. So "per frame" that would be 1KB or so (more for key frames, less for I frames). So they say it can be 0.1KB, so…

This is probably first-order-model[1] using keyframes. You send only one image each 2 seconds and the mesh of face 30 times per seconds.

Then use first-order-model to extrapolate 2 seconds of video from the keyframe.

Rinse, repeat.

Very doable. AMAZING!

The original first-order-model could not do 30 frames per second, but maybe this Nvidia model has some improvements.

1 - https://aliaksandrsiarohin.github.io/first-order-model-websi...

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#159

Post by gdubs »

polytely wrote: post #102
OrderlyTiamat wrote: post #12

> they have managed to reduce the required bandwidth for a video call by an order of magnitude. In one example, the required data rate fell from 97.28 KB/frame to a measly 0.1165 KB/frame – a reduction to 0.1% of required bandwidth. A nitpick, perhaps, but isn't that three orders of magnitude? We've already seen people use outlandish backgrounds in calls, now it's going to be possible to design similar outlandish vie…

I think you are right on the money with your thoughts on virtual avatars. I've already noticed this phenomenon cropping up in some niches. 1. the phenomenon of VTubers https://en.m.wikipedia.org/wiki/Virtual_YouTuber 2. in the virtual animal crossing late night show, Animal Talking, the presenter's (Gary Whitta) avatar doesn't really resemble how the presenter looks in real life https://en.m.wikipedia.org/wiki/Animal…

Jaron Lanier’s book on VR went in-depth on the importance of avatars, and experiences people had embodying different avatars — particularly in the early days of first-wave VR.

Re: Nvidia Uses AI to Slash Bandwidth on Video Calls

#160

Post by motoboi »

unicornporn wrote: post #7

Stills OK, it would be interesting to see it move. Risk for uncanny valley? Petapixel is a blog spam site btw. Why not go to the source that is linked in the post?

I suppose on can put boundaries in movement. If the face changes significantly, just send another keyframe.
Post reply on HN