Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The iPad was on Tailscale: a WebRTC debugging story

p2claw.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: The iPad was on Tailscale: a WebRTC debugging story

#21

Post by talkingtab »

Hunting good bugs is something every good software developer should experience. A good interview question is "tell me your favorite bug". Bugs are about reasoning, not intelligence. And I will take someone who can tell me what is wrong over what is correct any day. It requires a focus on getting things actually work.

I have two favorite bug stoies. The first is from a printout from the run of an IBM 360 assembly language program when I was just learning. Someone asked em why their program failed to run. I glanced quickly at the front page of the printout and it said "Too Long". So I told the person that was the problem. Something was too long. He looked at me very strangely, so I looked back at the page a little more closely, only to notice "Too Long" was in the name field of the person running the program. He was Vietnamese and his name was Too Long - literally. There is a powerful lesson (at least one) there.

The other happened when I was implementing some AppleTalk protocols - NBP to be exact. (Don't ask). I would capture the working packets then compare all the checksums, headers, constants, length fields in the packet my code generated and fix any problems. I was stuck on one failure. I just could not see any difference as I went through byte by byte, time after time. It was late and time to go home so I decided to print off each packet on paper and compare them later - certain I was missing something. The problem was instantly obvious. One printout took a page, the two pages. I had been appending junk data in the packet. Sigh

Re: The iPad was on Tailscale: a WebRTC debugging story

#22

Post by inigyou »

syllogistic wrote: post #15
happyopossum wrote: post #12

Earlier quoted context omitted.

> I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets. I’d venture to guess based on this outcome that fragmented UDP over IPv6 isn’t really an ordinary occurrence. Given the preponderance of HTTPS traffic, the aversion to fragmentation in IPv6, and the weird corner case of there being a hardcoded packet size in webrtc, it’s reasonable to assume t…

Would agree it's uncommon in general traffic. Rare conditions [webrtc-rs, 1280 class tunnel / tailscale, and ipv6 pair] but deadly when they are met since every connection silently fails. That's what made it worth chasing down for 2 weeks [and good for sleuthing :)].

It's a corner case of ordinary traffic, since all TCP apps and most UDP apps adapt to PMTU, but fragmentation is there for those that don't. It's not like something you can only get by generating malicious traffic intentionally.

Welcome to networking mistakes, I guess. I can't remember the specifics but I once encountered a router that would drop traffic that looked like encapsulated TCP at a certain offset, or something like that. They couldn't fix it because the behavior was hardwired. I knew of it because I worked with the firmware team.

Factorio discovered that UDP packets with a checksum of 0x0000 get dropped by some devices.

Re: The iPad was on Tailscale: a WebRTC debugging story

#24

Post by syllogistic »

Sean-Der wrote: post #19
cyanydeez wrote: post #16

Earlier quoted context omitted.

just wait till you try to send a data packet in webrtc that's too large in the browser. https://stackoverflow.com/questions/35381237/webrtc-data-cha... last I checked, all browsers silently fail if it's too big.

This should be fixed! I added this in Pion here[0] and I remember testing against Chrome + FireFox and it seemed to work great! [0] https://github.com/pion/webrtc/commit/e4ff415b2bff31382bdb80...

Good to know, thanks!

Though maybe I’ll keep my old limits for old browser compatibility.

Re: The iPad was on Tailscale: a WebRTC debugging story

#26

Post by sulam »

syllogistic wrote: post #11
Sean-Der wrote: post #9

Amazing debugging, I loved reading that. HN doesn't get enough good posts like this anymore :) If https://github.com/pion/sctp/issues/12 had happened (not just in Pion but across all implementations) this could have been fixed years ago. The hardcoding we all settle for is tragic.

Author here, thank you, that means a lot coming from you. Pion was the prior art I pointed the webrtc-rs maintainers at. And pion/sctp#12 is super relevant. A known, proposed fix years before we hit it. "The hardcoding we all settle for" might be the epigraph for the whole incident. webrtc-rs invited a PR for the configurable-MTU + better default half [webrtc-rs/webrtc#806] to unblock folks today. Whether PMTUD gets…

The debugging was interesting. I'm just going to have to learn to live with this I feel like, but the very LLM-ish language in the blog post was kind of annoying.

Re: The iPad was on Tailscale: a WebRTC debugging story

#27

Post by win311fwg »

talkingtab wrote: post #21

Hunting good bugs is something every good software developer should experience. A good interview question is "tell me your favorite bug". Bugs are about reasoning, not intelligence. And I will take someone who can tell me what is wrong over what is correct any day. It requires a focus on getting things actually work. I have two favorite bug stoies. The first is from a printout from the run of an IBM 360 assembly lang…

> A good interview question is "tell me your favorite bug".

I wouldn't have a clue how to recall any details about the bugs I've seen. I don't put much emphasis on past events. Looking forward is what I find to be a far more valuable use of my mental energy. I have vague recollections of debugging some doozies, but that is where the recall ends. It is clearly something you are passionate about, which no doubt keeps it something front of mind for you, but for many of it is just part of the job; like asking someone at McDonald's how their favourite burger flip landed.

You could say that I'm not the one of the for the job, which is a fair take, but if we reason through this some more, would we not conclude that there is no such thing as a good canned interview question? Given that no two people are the same, good interview questions can only be established in the context of who is being interviewed.

Re: The iPad was on Tailscale: a WebRTC debugging story

#28

Post by hylaride »

wmf wrote: post #20
hylaride wrote: post #8

I'm having flashbacks to 1990s-era PPPoE, where the slightly smaller MTU had issues with some server OS's that had TCP/IP stacks that didn't support or ignored MTUs smaller than 1500 bytes and bulk data transfers would get messed up. I don't remember which ones, but it was some commercial UNIX.

Weren't T1s running 576 MTU back then?

IIRC, T-lines had no inherent MTU. The MTU is determined by whatever layer-2 encapsulation ran over the T1:

PPP → default MTU 1500 Cisco HDLC → 1500 Frame Relay → typically 1500+ (often configurable higher)

So a typical IP MTU on a T1 link was 1500 bytes, same as Ethernet — chosen partly so packets could traverse Ethernet ↔ T1 boundaries without fragmenting.

576 is from RFC 791 (IPv4):

* Every IP host must be able to reassemble a datagram of at least 576 bytes.

* 576 became the conventional default MTU for "non-local" destinations — i.e., when a host didn't know the path MTU and wanted a value virtually guaranteed not to be fragmented anywhere. (576 − 20 IP − 20 TCP = 536 bytes of payload, the classic TCP default MSS.)

You'd also see 576 as a common default on dial-up/PPP links and X.25, which is probably the source of the association — not T1.

But re-assemble didn't necessarily mean transmit. My understanding (and this is quoting from memory from over 20 years ago) is that some commercial UNIXs from eons ago didn't ever really test dialup or other other such settings as they were often in more commercial settings and other protocols were often used before everything converged on IP. I'm sure these were also unpatched machines.

It was just annoying enough where some random connections didn't work very well.

Re: The iPad was on Tailscale: a WebRTC debugging story

#29

Post by lostmsu »

syllogistic wrote: post #3

Author here. This started as a blank page on one device and ended two weeks later at the intersection of two bugs: webrtc-rs hardcodes INITIAL_MTU=1228 [never updated, no path probing, retransmits at the same size forever], and Tailscale's packet filter classifies any IPv6 packet with a Fragment header as unknown protocol, so the default deny fires. On every platform, counted under reason="acl". Neither is unreasonab…

Just tried ping -s 1400 over Yggdrasil (IPv6 overlay) and it works. Reinforces my overlay choice.

Getting screwed by browsers though because their WebRTC implementations completely ignore Yggdrasil addresses.

Re: The iPad was on Tailscale: a WebRTC debugging story

#30

Post by bastard_op »

Developers should spend more time being network engineers before writing network code. I saw this title and my first thought was, "What, mtu?"

If you've ever had to support vpn's in an enterprise securing businesses with ipsec or sslvpn with tunnel overhead, you've run into mtu issues. Some apps/protocols or firewalls misbehave, devs/engineers didn't read the memo from 20 years ago in rfc form how ipv6 mtu's work (and missed v4 to boot, lucking out with 20 more years of someone else fixing it).

Not Tailscale or Cisco or in between are immune to mtu issues in vpn or networking.

Post reply on HN