Video Live Streaming: Notes on RTMP, HLS, and WebRTC
1–10 of 84 posts
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#2There is a new Low Latency HLS protocol: https://developer.apple.com/documentation/http_live_streamin... that claims can get down to 300ms latency.
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#3There is a new Low Latency HLS protocol: https://developer.apple.com/documentation/http_live_streamin... that claims can get down to 300ms latency.
Low latency HLS is creating partial segments by bucketing 200ms of frames instead of 6s segments in standard HLS. Whereas in webrtc, the endpoint is sending the frame as soon as it is ready.
The apples apples comparison here is 0ms (in webrtc, no send side buffering) vs 200ms (in low latency HLS) or 6s (in standard HLS). This is independent of latency of the endpoint from CDN or source.
Another distinction is playback wait time, i.e., how quickly upon joining can it start rendering video.
I’m assuming the full reference picture (typically, an IFrame or a golden frame depending on the codec) in low latency HLS is only available at the start of each 6s segment and not in partial segments. So joining a live stream, the receiving endpoint would have to wait at most 6s before rendering.
Similarly in webrtc, it’s up to the system to generate a reference frame at regular intervals, as low as every second. Or to do it reactively, a receiving endpoint can ask the sender to send a new reference picture. This is done via a Full intra request, the wait time can be as quick as 1.5 times of the round trip time (as new codecs can generate a new iframe instantaneously upon receiving a request). There’s a slight cpu penalty for this which means that the sender getting too many full intra requests may typically throttle the response to 1s.
So Apples to Apples comparison for wait time would be up to 1s for webrtc vs 6s for HLS.
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#4WebRTC is not the future of low latency live streaming... at least not outside of video conferencing. It's incredibly complex as a specification, has limitations and numerous issues that set limits in how scalable it can be. Conversely, for HTTP segment based formats like HLS and DASH have limits due to their design (and that of HTTP) which set how low latency can actually go.
Where's the future? A likely candidate may come out of the "Media over QUIC" work in the IETF, which already has several straw man protocols in real world use (Meta's RUSH, Twitch's WARP). It'll be a few more years before we see a real successor, but whatever it is will likely be able to supersede both WebRTC and HLS/DASH where QUIC and/or WebTransport is available.
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#5There is a new Low Latency HLS protocol: https://developer.apple.com/documentation/http_live_streamin... that claims can get down to 300ms latency.
Low latency HLS is creating partial segments by bucketing 200ms of frames instead of 6s segments in standard HLS. Whereas in webrtc, the endpoint is sending the frame as soon as it is ready. The apples apples comparison here is 0ms (in webrtc, no send side buffering) vs 200ms (in low latency HLS) or 6s (in standard HLS). This is independent of latency of the endpoint from CDN or source. Another distinction is playbac…
real world around:
HLS: 10-30s
LL-HLS: 2-6s
WebRTC: 0-2s. add couple more startup seconds depending slow is your signaling
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#6There is a new Low Latency HLS protocol: https://developer.apple.com/documentation/http_live_streamin... that claims can get down to 300ms latency.
this will be the winner purely because Apple devices are the lowest common denominator. That is, one must support iOS and if iOS only supports LL-HLS but other devices optionally support LL-HLS then LL-HLS is the winner.
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#7There is a new Low Latency HLS protocol: https://developer.apple.com/documentation/http_live_streamin... that claims can get down to 300ms latency.
[post author here]
In addition to what vr000m said above, I'll just add that when you make HLS chunks smaller, you're reducing the leverage you get from HLS's core design decisions. I tried to cover some of this in the post.
One way to think about this intuitively is that HLS and WebRTC are opposite ends of one important trade-offs axis.
HLS is about delivering media streams in a way that scales as cost-effectively as possible.
WebRTC is about delivering media frames at the lowest possible latency.
These are very different goals, and given current infrastructure and standards it's not possible to have your cake and eat it too, here. That may change in the future as low-latency video becomes more and more important. QUIC, for example, is a new approaches to building out a full stack that works around some of the fundamental tradeoffs that exist today.
The result is that pushing HLS segments down to 200ms is not at all a clear win. We'll see what happens as HLS implementations improve. And I should say that my brain has been warped by working on UDP/RTP stuff for a long time. But my bet is that using 200ms HLS segments is, for most real-world users, going to make HLS worse in every way than WebRTC would be, for the same use cases. (That's definitely true today with the early implementations of LLHLS.)
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#8There is a new Low Latency HLS protocol: https://developer.apple.com/documentation/http_live_streamin... that claims can get down to 300ms latency.
Low latency HLS is creating partial segments by bucketing 200ms of frames instead of 6s segments in standard HLS. Whereas in webrtc, the endpoint is sending the frame as soon as it is ready. The apples apples comparison here is 0ms (in webrtc, no send side buffering) vs 200ms (in low latency HLS) or 6s (in standard HLS). This is independent of latency of the endpoint from CDN or source. Another distinction is playbac…
An HLS segment can carry any number of GOPs. A GOP length of FPS/2 or FPS/4 will get you an I-frame pretty quickly allowing the GOP to be decoded. MPEG-DASH can do the same IIRC. So there doesn't need to be a segment length delay in playback and typically is not.
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#9WebRTC is not the future of low latency live streaming... at least not outside of video conferencing. It's incredibly complex as a specification, has limitations and numerous issues that set limits in how scalable it can be. Conversely, for HTTP segment based formats like HLS and DASH have limits due to their design (and that of HTTP) which set how low latency can actually go. Where's the future? A likely candidate m…
I am heavily biased toward WebRTC. Here is my take on it though!
> It's incredibly complex as a specification
What is complex about it? I can go and read the IETF drafts, webrtcforthecurious.com, https://github.com/adalkiran/webrtc-nuts-and-bolts and multiple implementations.
QUIC/WebTransport seems simple because it doesn't address all the things WebRTC does.
> has limitations and numerous issues that set limits in how scalable it can be
https://phenixrts.com/en-us/ does 500k viewers. I don't think anything about WebRTC makes it unscalable.
-----
IMO the future is WebRTC.
* Diverse users makes the ecosystem rich. WebRTC supports Conferencing, embedded, P2P/NAT Traversal, remote control... Every group of users has the made the ecosystem a little better.
* Client code is minimal. For most users they just need to exchange Session Descriptions and they are done. You then have additional APIs if you need to change behaviors. Other streaming protocols expect you to put lots of code client side. If you want to target lots of platforms that is a pretty big burden.
* Lots of implementations. C, C++, Python, Go, Typescript
* The new thing needs to be substantially better. I don't know what n is, but it isn't enough to just be a little better then WebRTC to replace it.
Re: Video Live Streaming: Notes on RTMP, HLS, and WebRTC
#10WebRTC is not the future of low latency live streaming... at least not outside of video conferencing. It's incredibly complex as a specification, has limitations and numerous issues that set limits in how scalable it can be. Conversely, for HTTP segment based formats like HLS and DASH have limits due to their design (and that of HTTP) which set how low latency can actually go. Where's the future? A likely candidate m…
Media over QUIC is interesting. For RTP or peer to peer QUIC, there is more work to be done. But you will end up engineering many of the same things as webrtc suit of protocols (ICE -- STUN, TURN, MULTIPLEXING, etc).
QUIC and webtransport can definitely already do DASH/HLS without some of the protocol complexity by using the QuicStreams (but to use QUICs underlying features, DASH/HLS need to change as well).
Some of us wrote a position statement in 2017, see https://datatracker.ietf.org/doc/html/draft-rtpfolks-quic-rt.... There are new documents around media ingest being proposed currently.