Video Live Streaming: Notes on RTMP, HLS, and WebRTC • Hacker News Board

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#51

Post by solar-ice » Mon, Jun 06, 2022, 9:18 PM UTC

vr000m wrote: post #42

Sean-Der wrote: post #35

Earlier quoted context omitted.

1) I don't think WebRTC has a specific point. Lots of users came together with their use cases and was designed by consensus. WebRTC can (and does) have toggles around latency/buffering. 2.) I am not aware of a way you can no keyframes, but be decodable at anytime. I just have done it 'HLS Style' or WebRTC 1:1. Curious if anyone else has different solutions.

Most common browser implementations use an Open GOP. That means an IFrame is implemented when needed. On scene change or when there’s high motion. Only naive implementations would burst an IFrame on to the network, most pace them. And if needed, you could split your iframe into several frame intervals and decode them without creating a burst by bit rate. Actually a lot of webrtc implementations use 1s or 2s GOP lengt…

That is not what open GOP means. Open GOP means pictures can reference IDR frames other than the most recent one in decode order, and is a pain in the ass for various reasons, but is technically more efficient. You're referring to a dynamic GOP.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#52

Post by difosfor » Mon, Jun 06, 2022, 9:24 PM UTC

We're using DASH and experimenting with LL-HLS to get reliable and affordable (using CDNS) end to end latency of 3 seconds with which you can already interact live with your audience without too much trouble. Latencies down to about 1 second are also feasible already at the cost of some client side buffering. So I would have expected to see more information about DASH than HLS here. It's usable on all platforms except iPhones and Apple TV where we're forced to use HLS at higher Latencies for now and hopefully later also DASH or LL-HLS once that's reliably usable. See: https://liveryvideo.com

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#53

Post by pthatcherg » Mon, Jun 06, 2022, 9:34 PM UTC

fiestajetsam wrote: post #4

WebRTC is not the future of low latency live streaming... at least not outside of video conferencing. It's incredibly complex as a specification, has limitations and numerous issues that set limits in how scalable it can be. Conversely, for HTTP segment based formats like HLS and DASH have limits due to their design (and that of HTTP) which set how low latency can actually go. Where's the future? A likely candidate m…

It's been almost 3 years since I first presented on WebTransport + WebCodecs: https://youtu.be/VD5GBLBiSxo

Live streaming was a motivating example for both of those, as you can tell from the video. And both of them grew out of our efforts to make WebRTC better for live streaming.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#54

Post by dbrueck » Mon, Jun 06, 2022, 9:35 PM UTC

Great article, thank you! Minor nit about footnote #10 (the very bad network failure mode): it really depends on where the bottleneck is and how the client has been implemented to react - there's a huge spectrum of client implementations out there, ranging from nearly dumb to those having wizard level heuristics.

If it's the user's last mile connection (between e.g. their home and their ISP), then the big HLS/DASH/etc buffer translates into a lot of time to react. So clients have the option to shift quite low - and do so quickly - if there are some very low bandwidth options, in theory even switching down to an audio-only or nearly-audio-only stream if one is provided, and can also choose to be optimistic/aggressive to resume playback as soon as one full chunk is downloaded - and some implementations will even resume playback when less than a full chunk is downloaded. The client side logic has a lot of latitude here to balance fast start/resume times vs sustaining playback.

When the bottleneck or failure is elsewhere, HLS can be incredibly durable. For extremely high profile events, for example, there are typically multiple CDNs involved, multiple sources going to independent encoders, etc. So an HLS/DASH client might talk to many different servers on a given CDN, as well as servers on alternate CDNs, and even grab what amount to being different copies of the stream spit out by different encoders. It's not uncommon for a client to be testing different CDN endpoints throughout playback to migrate away from congestion automatically.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#55

Post by dbrueck » Mon, Jun 06, 2022, 9:42 PM UTC

jzer0cool wrote: post #23

In each of these approaches will any of these (or something else) also provide approaches to getting the stream saved for viewing again later on a remote server? I'm guessing I need some client which also connects to the stream and save which can then be served again. Or is still a solution already baked in available?

From a technical perspective, that's more or less baked into HLS: these are just discrete files being served over HTTP, so you can save the files to disk along with the manifest file and you have the content.

In practice, there is a little more to it. For example, when DRM is enabled, you need some way to preserve the decryption keys. And for live content, the manifest file usually just tells the client about a sliding window of files, so you need a tiny bit of additional client side logic to pay attention to this fact.

One cool thing about DASH/HLS is that you can do some pretty complex mixing of content - you can build a traditional TV-channel like experience that mixes live and prerecorded content, you can replace and inject ads, you can make live content immediately available for on-demand playback, etc.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#56

Post by vdnkh » Mon, Jun 06, 2022, 9:46 PM UTC

Twitch latency in low-latency mode is ~2s (you can check for yourself in the settings cog menu -> advanced -> video stats) because of "prefetch segments", which are delivered via HTTP 1.1 streaming body responses. The client requests this segment and long polls while frames are delivered as they're being encoded. It's simpler and cheaper than WebRTC or Apple's low-latency HLS parts, and scales to hundreds of thousands of viewers. The downside is that it's not part of the HLS specification so support for it needs to be bespoke, but it is a proven technology that wasn't covered here (and to my knowledge, the most widely used low latency HTTP solution by # of users). Twitch has been using this for years.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#57

Post by pavlov » Mon, Jun 06, 2022, 9:57 PM UTC

rexreed wrote: post #48

vr000m wrote: post #47

Earlier quoted context omitted.

Pavlov’s comment is correct. I came to add that soon the stream can be stored on customer’s own S3. Ergo, you’d be able to do a call in real-time, store it on your S3 account and make it available for streaming. On your own S3, this would be a multipart upload.

If we use Vimeo for video hosting would there be some way to automatically post to Vimeo or some other way to get the file up there and available for immediate viewing? Other approaches besides S3?

If Vimeo offers an ingest API using either RTMP(S) or HLS, that would be one way to get the stream from Daily directly to them without any extra processing step in between.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#58

Post by saurik » Mon, Jun 06, 2022, 10:15 PM UTC

sumy23 wrote: post #38

Sean-Der wrote: post #35

Earlier quoted context omitted.

1) I don't think WebRTC has a specific point. Lots of users came together with their use cases and was designed by consensus. WebRTC can (and does) have toggles around latency/buffering. 2.) I am not aware of a way you can no keyframes, but be decodable at anytime. I just have done it 'HLS Style' or WebRTC 1:1. Curious if anyone else has different solutions.

1) WebRTC and RTP both have RT in their name. RT stands for real-time. If I recall correctly, the only buffer WebRTC has is the jitter buffer, which is used for packet ordering, not for ensuring that enough has buffered to handle bitrate spikes. 2) Yes, you either need a high keyframe interval or some type of out-of-band signaling framework to generate keyframes. WebRTC uses RTCP. A good question is why does WebRTC f…

1) That is a specific implementation, and has nothing to do with the protocol, which certainly doesn't define a "jitter buffer". People routinely use RTMP--which also has RT in the name--to transfer content to streaming services with massive buffers at every step in the pipeline.

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#59

Post by [deleted] » Mon, Jun 06, 2022, 11:10 PM UTC

fiestajetsam wrote: post #4

WebRTC is not the future of low latency live streaming... at least not outside of video conferencing. It's incredibly complex as a specification, has limitations and numerous issues that set limits in how scalable it can be. Conversely, for HTTP segment based formats like HLS and DASH have limits due to their design (and that of HTTP) which set how low latency can actually go. Where's the future? A likely candidate m…

[deleted]

Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC

#60

Post by Karrot_Kream » Mon, Jun 06, 2022, 11:19 PM UTC

vr000m wrote: post #50

Karrot_Kream wrote: post #34

Earlier quoted context omitted.

Heh, welcome to the world of livestreaming media. The reason why it's hard to create this kind of simple "stream in, stream out" abstraction is because most IP Voice/Video stacks are architected very differently than stateless net protocols that are popular today. IP streaming generally works by: 1. A signaling layer that helps setup the connection metadata (a layer where the sender can say they're the sender, that t…

All the listed protocols came after HTTP. RTSP, SIP borrowed heavily (albeit badly in retrospect) from HTTP. I do not have all the historical context (early 90s), but for WebRTC, the idea was to not define any new protocol(s) or do a clean slate design. but rather to just agree on the flavors of the various protocols, and then to universally implement those. We already had SDP, RTSP, RTP, SAP, etc. And the idea was t…

I know they came later, so I'm still confused why RTSP and SIP weren't implemented atop HTTP. I realize that RTSP and SIP can push server to client, but there's ways around that, though perhaps long polling and Websockets weren't conceivable when RTSP and SIP were invented. I mean, in a pinch, I have an HTTP server serving a folder where SDP files are generated, and I've written clients that just look for a well-known SDP file and use that to consume an RDP stream. It's a ghetto form of "signaling" that I love using when doing experiments (not suitable for production for various reasons obvious to you I imagine.)

I'm not saying WebRTC had poor design decisions or anything. I think it was very smart for WebRTC to reuse SDP, RDP, etc so the same libraries and pipelines could keep working with minimal changes. It also means very little new learning for folks familiar with the rest of the stack.

> For realtime media, if we are able combine “locator:identifier” issue, we will be able to make media and signaling work inband.

+1000. I think RTSP+TCP is a decent way to do in-band signaling and media, and RTMP defines strict ways to send both anyway.