Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The iPad was on Tailscale: a WebRTC debugging story

p2claw.com

Post reply on HN
Print view View original

11–20 of 40 posts

Re: The iPad was on Tailscale: a WebRTC debugging story

#11

Post by syllogistic »

Sean-Der wrote: post #9

Amazing debugging, I loved reading that. HN doesn't get enough good posts like this anymore :) If https://github.com/pion/sctp/issues/12 had happened (not just in Pion but across all implementations) this could have been fixed years ago. The hardcoding we all settle for is tragic.

Author here, thank you, that means a lot coming from you. Pion was the prior art I pointed the webrtc-rs maintainers at. And pion/sctp#12 is super relevant. A known, proposed fix years before we hit it.

"The hardcoding we all settle for" might be the epigraph for the whole incident. webrtc-rs invited a PR for the configurable-MTU + better default half [webrtc-rs/webrtc#806] to unblock folks today. Whether PMTUD gets implemented will be interesting to see.

Re: The iPad was on Tailscale: a WebRTC debugging story

#12

Post by happyopossum »

inigyou wrote: post #5

I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets. It is impossible to parse the UDP or TCP port number out of a fragment. This is surely the reason the ACL module entirely rejects them. TCP will adjust it's segment size based on PMTUD so as to not require fragmentation. This is why it hasn't been noticed so far. But fragmented UDP packets are a…

> I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets.

I’d venture to guess based on this outcome that fragmented UDP over IPv6 isn’t really an ordinary occurrence. Given the preponderance of HTTPS traffic, the aversion to fragmentation in IPv6, and the weird corner case of there being a hardcoded packet size in webrtc, it’s reasonable to assume that this is a corner case.

A good one to be aware of, but not common.

Re: The iPad was on Tailscale: a WebRTC debugging story

#14

Post by cyberax »

Another fun happy iOS story: we were launching our app a year ago, with a self-imposed deadline. As usual, tons of bugs were being fixed in the last moment.

And then our authentication stopped working on simulated iOS devices (while still working on the real devices!). After hours of frantic debugging and staring at Wireshark dumps, I found the issue: HTTP3 and QUIC. Apparently, the simulated stack was not tracking the MTU correctly and was trying to send 1506-byte UDP packets.

The "fix" was to add deny rules for UDP ports 80/443 to our firewall.

Re: The iPad was on Tailscale: a WebRTC debugging story

#15

Post by syllogistic »

happyopossum wrote: post #12
inigyou wrote: post #5

I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets. It is impossible to parse the UDP or TCP port number out of a fragment. This is surely the reason the ACL module entirely rejects them. TCP will adjust it's segment size based on PMTUD so as to not require fragmentation. This is why it hasn't been noticed so far. But fragmented UDP packets are a…

> I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets. I’d venture to guess based on this outcome that fragmented UDP over IPv6 isn’t really an ordinary occurrence. Given the preponderance of HTTPS traffic, the aversion to fragmentation in IPv6, and the weird corner case of there being a hardcoded packet size in webrtc, it’s reasonable to assume t…

Would agree it's uncommon in general traffic. Rare conditions [webrtc-rs, 1280 class tunnel / tailscale, and ipv6 pair] but deadly when they are met since every connection silently fails. That's what made it worth chasing down for 2 weeks [and good for sleuthing :)].

Re: The iPad was on Tailscale: a WebRTC debugging story

#16

Post by cyanydeez »

syllogistic wrote: post #3

Author here. This started as a blank page on one device and ended two weeks later at the intersection of two bugs: webrtc-rs hardcodes INITIAL_MTU=1228 [never updated, no path probing, retransmits at the same size forever], and Tailscale's packet filter classifies any IPv6 packet with a Fragment header as unknown protocol, so the default deny fires. On every platform, counted under reason="acl". Neither is unreasonab…

just wait till you try to send a data packet in webrtc that's too large in the browser. https://stackoverflow.com/questions/35381237/webrtc-data-cha...

last I checked, all browsers silently fail if it's too big.

Re: The iPad was on Tailscale: a WebRTC debugging story

#17

Post by syllogistic »

cyanydeez wrote: post #16
syllogistic wrote: post #3

Author here. This started as a blank page on one device and ended two weeks later at the intersection of two bugs: webrtc-rs hardcodes INITIAL_MTU=1228 [never updated, no path probing, retransmits at the same size forever], and Tailscale's packet filter classifies any IPv6 packet with a Fragment header as unknown protocol, so the default deny fires. On every platform, counted under reason="acl". Neither is unreasonab…

just wait till you try to send a data packet in webrtc that's too large in the browser. https://stackoverflow.com/questions/35381237/webrtc-data-cha... last I checked, all browsers silently fail if it's too big.

Yes! that's the "other reconstruct" I mention on the post. maxMessageSize at least appears in SDP and getStats. We ended up patching both at our client to be safe [800 bytes and 16kb respectively].

Re: The iPad was on Tailscale: a WebRTC debugging story

#19

Post by Sean-Der »

cyanydeez wrote: post #16
syllogistic wrote: post #3

Author here. This started as a blank page on one device and ended two weeks later at the intersection of two bugs: webrtc-rs hardcodes INITIAL_MTU=1228 [never updated, no path probing, retransmits at the same size forever], and Tailscale's packet filter classifies any IPv6 packet with a Fragment header as unknown protocol, so the default deny fires. On every platform, counted under reason="acl". Neither is unreasonab…

just wait till you try to send a data packet in webrtc that's too large in the browser. https://stackoverflow.com/questions/35381237/webrtc-data-cha... last I checked, all browsers silently fail if it's too big.

This should be fixed!

I added this in Pion here[0] and I remember testing against Chrome + FireFox and it seemed to work great!

[0] https://github.com/pion/webrtc/commit/e4ff415b2bff31382bdb80...

Re: The iPad was on Tailscale: a WebRTC debugging story

#20

Post by wmf »

hylaride wrote: post #8

I'm having flashbacks to 1990s-era PPPoE, where the slightly smaller MTU had issues with some server OS's that had TCP/IP stacks that didn't support or ignored MTUs smaller than 1500 bytes and bulk data transfers would get messed up. I don't remember which ones, but it was some commercial UNIX.

Weren't T1s running 576 MTU back then?
Post reply on HN