Live data from Hacker News

The first Media over QUIC CDN: Cloudflare

moq.dev

61–70 of 125 posts

Re: The first Media over QUIC CDN: Cloudflare

#61
post #54

Earlier quoted context omitted.

EDIT: Sorry I just noticed this was directed to Cloudflare. They're using the same architecture as Cloudflare Realtime, their WebRTC offering. `relay.moq.dev` currently uses GeoDNS to route to the closest edge. I'd like to use anycast like Cloudflare (and QUIC's preferred_address), but cloud offerings for anycast + UDP are limited. The relays nodes currently form a mesh network and gossip origins between themselves.…

Home much success have you have with GeoDNS? We've seen it fail when users are using privacy respecting resolvers like 1.1.1.1. It gets the continent right but fails on city/state level.

It works well right now because there's only one edge per continent. But if I had traffic, anycast is definitely the way to go.

Re: The first Media over QUIC CDN: Cloudflare

#62

Earlier quoted context omitted.

QUIC is already quite widely used! We see close to 10% of HTTP requests using HTTP/3: https://radar.cloudflare.com/adoption-and-usage As for the NAT problem, that's mainly an issue for peer-to-peer scenarios. If you have a publicly addressable server at one end, you don't need all of the complications of a full ICE stack, even for WebRTC. For cases where you do need TURN (e.g. for WebRTC with clients that may be on n…

Thanks for the answer! Unfortunately the NAT problem is more common than you think :( Lot's of corporate networks use full cone NAT (I know ours does), and so does AWS (if you don't have a public IP, but go through igw), so some sort of NAT punchthrough seems to be necessary for WebRTC. I wonder if WebTransport has its own solution to the problem. But I guess you can always rely on turn - by the way, does MoQ have so…

I answered in another reply, but client -> server protocols like TCP and QUIC don't have an issue traversing NATs. The biggest problem you'll run into are corporate firewalls blocking UDP, but hopefully HTTP/3 adoption helps that (UDP :443, same as WebTransport).

Re: The first Media over QUIC CDN: Cloudflare

#63
post #44

Semi-related and just FYI for Firefox users who visit Cloudflare-hosted, HTTP/3-using sites: https://bugzilla.mozilla.org/show_bug.cgi?id=1979683

Limited to macOS? Does not reproduce in FF 141 and 142 on Windows.

No, I have also observed it in Firefox (via Flatpak) on Fedora Linux 42. When I filed the original GH issue that webcompat-bot turned into this Bugzilla item (https://github.com/webcompat/web-bugs/issues/168913), my full report didn't make it into Bugzilla.

Re: The first Media over QUIC CDN: Cloudflare

#64
post #3

Hi! Cloudflare MoQ dev here, happy to answer questions! Thanks for the award, kixelated. xD

Hi, I've got one.

Does your team have any concrete plans to reduce the TCP vs. QUIC diff with respect to goodput [1]? The linked paper claims seeing up to a 9.8% video bitrate reduction from HTTP/2 (TCP) to HTTP/3 (QUIC). Obviously, MoQ is based on a slightly different stack, so the results don't exactly generalize. I can imagine the problems are similar, though.

(I find this stuff fascinating, as I spent the last few months investigating the AF_XDP datapath for MsQuic as part of my master's thesis. I basically came to the conclusion that GSO/GRO is a better alternative and that QUIC desperately needs more hardware offloads :p)

[1]: https://arxiv.org/pdf/2310.09423

Re: The first Media over QUIC CDN: Cloudflare

#65
post #49

I tested the demo at https://moq.dev/publish/ and it's buttery as hell. Very impressive. Thanks for the great technology! Watching the Big Buck Bunny demo at https://moq.dev/watch/?name=bbb on my mobile phone leaves a lot of horizontal black lines. (Strangely, it is OK on my PC despite using the same Wi-Fi network.) Is it due to buffer size? Can I increase it client-side, or should it be done server-side? Also, thank…

Horizontal black lines? Dunno what that could be about, we render to a element which is resized to match the source video and then resized again to match the window with CSS.

I have got same issue with the black lines

Re: The first Media over QUIC CDN: Cloudflare

#67

Semi-related and just FYI for Firefox users who visit Cloudflare-hosted, HTTP/3-using sites: https://bugzilla.mozilla.org/show_bug.cgi?id=1979683

Hmm…do you have the in-browser DNS over HTTPS resolver enabled? I personally can't reproduce this, but I'm using DoH with 1.1.1.1.

I've noticed that both Chrome and Firefox tend to have less consistent HTTP/3 usage when using system DNS instead of the DoH resolver because a lot of times the browser is unable to fetch HTTPS DNS records consistently (or at all) via the system resolver.

Since HTTP/3 support on the server has to be advertised by either an HTTPS DNS record or a cached Alt-Svc header from a previous successful HTTP/2 or HTTP/1.1 connection, and the browsers tend to prefer recycling already open connections rather than opening new ones (even if they would be "upgraded" in that case), it's often much trickier to get HTTP/3 to be used in that case. (Alt-Svc headers also sometimes don't cache consistently, especially in Firefox in my experience.)

Also, to make matters even worse, the browsers, especially Chrome, seem to automatically disable HTTP/3 support if connections fail often enough. This happened to me when I was using my university's Wi-Fi a lot, which seems to block a large (but inconsistent) amount of UDP traffic. If Chrome enters this state, it stops using HTTP/3 entirely, and provides no reasoning in the developer tools as to why (normally, if you enable the "Protocol" column in the developer tools Network tab, you can hover over the listed protocol to get a tooltip explaining how Chrome determined the selected protocol was the best option available; this tooltip doesn't appear in this "force disabled" state). Annoyingly, Chrome also doesn't (or can't) isolate this state to just one network, and instead I suddenly stopped being able to use HTTP/3 at home, either. The only actual solution/override to this is to go into about:flags (yes, I know it's chrome://flags now, I don't care) and make sure that the option for QUIC support is manually enabled. Even if it's already indicated as "enabled by default", this doesn't actually reflect the browser's true state. Firefox also similarly gives up on HTTP/3, but its mechanism seems to be much less "sticky" than Chrome's, and I haven't had any consistent issues with it.

To debug further: I'd first try checking to see if EncryptedClientHello is working for you or not; you can check https://tls-ech.dev to test that. ECH requires HTTPS DNS record support, so if that shows as working, you can ensure that your configuration is able to parse HTTPS records (that site also only uses the HTTPS record for the ECH key and uses HTTP/1.1 for the actual site, so it's fairly isolated from other problems). Next, you can try Fastly's HTTP/3 checker at https://http3.is which has the benefit of only using Alt-Svc headers to negotiate; this means that the first load will always use HTTP/2, but you should be able to refresh the page and get a successful HTTP/3 connection. Cloudflare's test page at https://cloudflare-quic.com uses both HTTPS DNS records and an Alt-Svc header, so if you are able to get an HTTP/3 connection to it first try, then you know that you're parsing HTTPS records properly.

Let me know how those tests perform for you; it's possible there is an issue in Firefox but it isn't occurring consistently for everyone due to one of the many issues I just listed.

(If anyone from Cloudflare happens to be reading this, you should know that you have some kind of misconfiguration blocking https://cloudflare-quic.com/favicon.ico and there's also a slight page load delay on that page because you're pulling one of the images out of the Wayback Machine via https://web.archive.org/web/20230424015350im_/https://www.cl... when you should use an "id_" link for images instead so the Internet Archive servers don't have to try and rewrite anything, which is the cause of most of the delays you typically see from the Wayback Machine. (I actually used that feature along with Cloudflare Workers to temporarily resurrect an entire site during a failed server move a couple of years back, it worked splendidly as soon as I learned about the id_ trick.) Alternatively, you could also just switch that asset back to https://www.cloudflare.com/img/nav/globe-lang-select-dark.sv... since it's still live on your main site anyway, so there's no need to pull it from the Wayback Machine.)

I've spent a lot of time experimenting with HTTP/3 and it's weird quirks over the past couple of years. It's a great protocol, it just has a lot of bizarre and weirdly specific implementation and deployment issues.

Re: The first Media over QUIC CDN: Cloudflare

#68
post #44

Semi-related and just FYI for Firefox users who visit Cloudflare-hosted, HTTP/3-using sites: https://bugzilla.mozilla.org/show_bug.cgi?id=1979683

Limited to macOS? Does not reproduce in FF 141 and 142 on Windows.

It reproduces for me in FF 142 on Windows. When I first went to https://cloudflare-quic.com/ it said HTTP/3, but after a few hard refreshes it says HTTP/2 and hasn't gone back to 3

Re: The first Media over QUIC CDN: Cloudflare

#69
post #3

Hi! Cloudflare MoQ dev here, happy to answer questions! Thanks for the award, kixelated. xD

Hi, I've got one. Does your team have any concrete plans to reduce the TCP vs. QUIC diff with respect to goodput [1]? The linked paper claims seeing up to a 9.8% video bitrate reduction from HTTP/2 (TCP) to HTTP/3 (QUIC). Obviously, MoQ is based on a slightly different stack, so the results don't exactly generalize. I can imagine the problems are similar, though. (I find this stuff fascinating, as I spent the last fe…

QUIC implementations are definitely not tuned well in practice for 600Mbps flows on low latency, low loss networks, as the paper attests. But I don’t think almost any uses of video streaming fit that bill. Even streaming 4K video via Netflix or similar is tens of Mbps. In general if you don’t have loss or the need to rapidly establish connections, QUIC performance is not even theoretically better, let alone in practice.

P.S. if there’s a public link to your masters thesis - please post it! I’d love to read how that shook out, even if AF_XDP didn’t fit in the end.

Post reply on HN