How to stream media using WebRTC and FFmpeg, and why it's a bad idea • Hacker News Board

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#31

Post by zaroth » Sun, Jan 31, 2021, 2:35 AM UTC

mandis wrote: post #2

>And finally, we encounter a large issue without a good solution. In encoded videos, a key frame is a frame in the video that contains all the visual information needed to render itself without any additional metadata. These are much larger than normal frames, and contribute greatly to the bitrate. Ideally, there would be as a few keyframes as possible. However, when a new user starts consuming a stream, they need at…

I guess it comes down to latency requirements?

I would expect where latency isn’t a huge concern, the best user experience would be to start the new receiver back at the last keyframe and fill the buffer up to “present” so they can start watching instantly, and keep a few seconds in the buffer for stability.

In more latency critical streams where you still want the perception of instant video startup I suppose you would have to start at the last keyframe and then as soon as the next key frame came through you could just jump ahead.

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#32

Post by j1elo » Sun, Jan 31, 2021, 2:45 AM UTC

rlyshw wrote: post #3

What about the WebRTC part? The post ends at RTP out from FFMPEG. Maybe I’m supposed to know how to consume that with WebRTC but in my investigation it’s not at all straightforward... the WebRTC consumer needs to become aware of the stream through a whole complicated signaling and negotiation process. How is that handled after the FFMPEG RTP stream is produced?

The WebRTC part would indeed be convoluted.

First, you would need to encrypt the RTP packets with DTLS.

Then, you would need an SDP message generator, where you would include all sorts of info:

* Codec and tunings of video and audio streams.

* RTCP ports where you'll be listening from RTCP Receiver feedback, if any.

* The TLS keys used for encryption.

* Some fake ICE candidates that the other part can use to reach you.

Then provide this as an SDP Offer to the WebRTC API of the other side (i.e. the RTCPeerConnection if we're talking about a web browser), and receive in response an SDP Answer. You should then be able to parse this Answer because the other participant might have rejected some of the parameters you gave it in the Offer (e.g. it could be ready only for audio and reject your video). Or just ignore the Answer and hope that you know the other party so well that they won't reject any of the parameters you provided in the Offer.

Finally you would need to receive ICE candidates from the other party, and parse them in order to know where (what IP and port) to send your RTP packets (and RTCP Sender Reports, if any)

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#33

Post by dimes » Sun, Jan 31, 2021, 2:55 AM UTC

scottlamb wrote: post #30

mandis wrote: post #2

>And finally, we encounter a large issue without a good solution. In encoded videos, a key frame is a frame in the video that contains all the visual information needed to render itself without any additional metadata. These are much larger than normal frames, and contribute greatly to the bitrate. Ideally, there would be as a few keyframes as possible. However, when a new user starts consuming a stream, they need at…

(The quoted paragraph is no longer in the article. But I'm still curious about it.) > Therefore, the parameter -force_key_frames expr:gte(t,n_forced*4) is needed, which produces a key frame every 4 seconds. How often would you like it to be producing key frames? My video experience is mostly with security cameras, and the ones I've used produce an I-frame every 2 seconds by default. Their encoders don't seem to be re…

As a reference, I think Google Chrome sends a key frame every 90 seconds by default

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#34

Post by banana_giraffe » Sun, Jan 31, 2021, 3:15 AM UTC

scottlamb wrote: post #30

mandis wrote: post #2

>And finally, we encounter a large issue without a good solution. In encoded videos, a key frame is a frame in the video that contains all the visual information needed to render itself without any additional metadata. These are much larger than normal frames, and contribute greatly to the bitrate. Ideally, there would be as a few keyframes as possible. However, when a new user starts consuming a stream, they need at…

(The quoted paragraph is no longer in the article. But I'm still curious about it.) > Therefore, the parameter -force_key_frames expr:gte(t,n_forced*4) is needed, which produces a key frame every 4 seconds. How often would you like it to be producing key frames? My video experience is mostly with security cameras, and the ones I've used produce an I-frame every 2 seconds by default. Their encoders don't seem to be re…

A few random videos I pulled from youtube had a key frame on average every 4.5 seconds, ranging from 0.1 to 5.5 seconds apart, seems pretty consistent regardless of type of video, at least for the few I tried.

The worst I've seen in production was a poorly configured encoder that insert a keyframe exactly every 30 seconds, along with segments every 10 seconds, which surprisingly, caused some players to crash trying to find a keyframe.

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#35

Post by at_a_remove » Sun, Jan 31, 2021, 3:55 AM UTC

j1elo wrote: post #29

Note that this post doesn't really cover how to stream media using WebRTC. First and foremost, because WebRTC mandates the use of DTLS to encrypt the RTP flow, thus a plain RTP stream won't work. A more apt title would be "How to use FFmpeg to generate an encoded stream that happens to match the requirements for WebRTC". Still, thanks for the article; it is always interesting to see specific applications of the FFmpe…

This is one of my pet peeves with documentation. "When to" and "when not to" are essentially the wisdom to match the intelligence of a how-to.

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#36

Post by Orphis » Sun, Jan 31, 2021, 5:10 AM UTC

j1elo wrote: post #29

Note that this post doesn't really cover how to stream media using WebRTC. First and foremost, because WebRTC mandates the use of DTLS to encrypt the RTP flow, thus a plain RTP stream won't work. A more apt title would be "How to use FFmpeg to generate an encoded stream that happens to match the requirements for WebRTC". Still, thanks for the article; it is always interesting to see specific applications of the FFmpe…

Technically, it's SRTP with keys derived from the handshake of a DTLS connection though. That DTLS connection can be used for SCTP, the underlying protocol for WebRTC data channels.

So yeah, that won't work to stream to a WebRTC endpoint as you said!

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#37

Post by beagle3 » Sun, Jan 31, 2021, 8:17 AM UTC

Dylan16807 wrote: post #15

tatersolid wrote: post #4

Earlier quoted context omitted.

H.264 and most other modern codecs support “intra refresh” to avoid this problem, at the cost of a marginally higher bitrate overall. Think of this as a “rolling keyframe slice” which marches across the screen every few seconds. http://www.chaneru.com/Roku/HLS/X264_Settings.htm#intra-refr...

I would say intra refresh solves a different problem. You still have to wait for the intra refresh to cover the frame before you can start watching properly. That takes just as long as waiting for a keyframe, and needs slightly more bytes. The benefit of intra refresh is that you avoid having any particularly large frames. If you're using a sub-second buffer, then intra refresh makes your maximum frame size much smal…

You could do on most things, but it has to be part of the codec to actually work.

If you aren’t aware of this scheme, you can just wait until the next key frame. With IDR there is a lot more bookkeeping to do so you can figure out when every single needed pixel has been accounted for.

And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something.

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#38

Post by Dylan16807 » Sun, Jan 31, 2021, 11:00 AM UTC

beagle3 wrote: post #37

Dylan16807 wrote: post #15

Earlier quoted context omitted.

I would say intra refresh solves a different problem. You still have to wait for the intra refresh to cover the frame before you can start watching properly. That takes just as long as waiting for a keyframe, and needs slightly more bytes. The benefit of intra refresh is that you avoid having any particularly large frames. If you're using a sub-second buffer, then intra refresh makes your maximum frame size much smal…

You could do on most things, but it has to be part of the codec to actually work. If you aren’t aware of this scheme, you can just wait until the next key frame. With IDR there is a lot more bookkeeping to do so you can figure out when every single needed pixel has been accounted for. And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probabl…

> And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something.

I think that supports my point. It doesn't matter if the technique is explicitly listed in the codec or not. You need clients that will tolerate infinite P-frames, and they will work equally well whether you're using intra refresh techniques on h.264 or h.262

The dumbest possible renderer would have full support; it takes extra logic to get in the way and break it.

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#39

Post by mpfundstein » Sun, Jan 31, 2021, 1:06 PM UTC

j1elo wrote: post #29

Note that this post doesn't really cover how to stream media using WebRTC. First and foremost, because WebRTC mandates the use of DTLS to encrypt the RTP flow, thus a plain RTP stream won't work. A more apt title would be "How to use FFmpeg to generate an encoded stream that happens to match the requirements for WebRTC". Still, thanks for the article; it is always interesting to see specific applications of the FFmpe…

a lot of open source media docs are like this. one of the worst offenders is gstreamer. geez is that stuff uninformative.

but at least its awesome software for free so who am I to complain?

Re: How to stream media using WebRTC and FFmpeg, and why it's a bad idea

#40

Post by SirSavary » Mon, Feb 01, 2021, 6:41 PM UTC

wwweston wrote: post #12

mandis wrote: post #2

>And finally, we encounter a large issue without a good solution. In encoded videos, a key frame is a frame in the video that contains all the visual information needed to render itself without any additional metadata. These are much larger than normal frames, and contribute greatly to the bitrate. Ideally, there would be as a few keyframes as possible. However, when a new user starts consuming a stream, they need at…

Thanks for the easy summary. One thing to consider: for some IRL performances, it's not uncommon that if you arrive late, you might be seated at the timing discretion of an usher. I understand digital experiences may carry different expectations, but I could see building an experience around this, perhaps starting with audio-only and maybe even a countdown to a next keyframe event (every minute?) while a "please wait…

Incredible -- "Our digital usher is finding you a seat in the cloud" slapped on a screen might just save us months of planning and millions on infrastructure.