"Adventures with asymmetric routing and firewalls"[1] might provide some useful insite/information[1].
[1] : http://www.growse.com/2020/01/23/adventures-with-asymmetric-...
TCP connection timeout mystery
21–30 of 83 posts
Re: TCP connection timeout mystery
#22I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.
Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.
Re: TCP connection timeout mystery
#2399% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.
Edit: on reading a few more comments, I think this is probably all wrong...
The TLS Client hello is not that big (the client sent FIN is seq=518), and the server is only sending packets with SEQ=0. As others pointed out this likely means that the server that received the SYNs is not receiving the final ACK and data packets.
From what I can tell, the example IP is not broadly anycast. From my test hosts in Seattle, traceroute takes me trhough transit to San Jose, and then either
vl201.sjc-eq10-dist-1.cdn77.com or vl202.sjc-eq10-dist-1.cdn77.com and finally
169-150-221-147.bunnyinfra.net
I'm not sure how easy it is to run a traceroute with tcp with different flags. But if the OP can run a traceroute with only the SYN flag, and again with only the ACK flag, that might be pretty interesting. I suspect this is an issue inside BunnyCDN's network where packets from this user/network with SYN go to one server host, and with ACK go to another. Maybe there's an odd router somewhere that's routing these differently, but if they both make it to Bunny, they should both work.
With
$ traceroute --version
Modern traceroute for Linux, version 2.1.2
Copyright (c) 2016 Dmitry Butskoy, License: GPL v2 or any later
traceroute 169.150.221.147 -p 443 -q 1 -T -O ack
traceroute 169.150.221.147 -p 443 -q 1 -T -O syn
Yeah, that would be my bet too. Especially with a after 60 seconds things start to work, I think that's the timeout for windows to do PMTU Blackhole probing (which is painfully slow; iOS and I think MacOS do it much sooner; I think even Android has gotten around to doing it in a reasonable amount of time)
I've got a test site up that might work for the OP http://pmtud.enslaves.us/
But, if it's really only happening with BunnyCDN, it's possible that most of their routes are 1500 MTU clean (or have working path MTU) and only the routes to get to BunnyCDN aren't. Of course, a lot of popular services intentionally drop their advertised MTU and allowed outbound MTU to work around the many broken networks out there, so service X and Y works doesn't really mean the path is clean.
Re: TCP connection timeout mystery
#24Earlier quoted context omitted.
The article mentions that it happens both over HTTP and HTTPS. I'd ask OP to check if this is only affects a subset of their IPs from https://bunnycdn.com/api/system/edgeserverlist , or whether all of their IPs are affected using `curl --resolve bunnycdn-hosted-website.com:80:some-other-ip http://bunnycdn-hosted-website.com `.
Besides that, the author points out that the final handshake ACK never reaches the server and that packet is small, not going to go over the mtu.
Indeed, it's right there in the packet capture screenshot. The ack has payload length 0.
I've debugged a lot of TCP/IP issues over the years but this one has me scratching my head. The author has done reasonable troubleshooting: tried from different devices and operating systems, HTTP and HTTPS, over wired and WiFi, and to different destinations. The common denominator is the wired network.
It can't hurt to reduce the MTU, but I see nothing in the evidence presented that this is likely to be the cause.
I once had a destination firewall blocking packets from Linux but not OS X and it turned out to be that Linux was an early adopter of ECN and the destination firewall rejected any packets with the ECN bits set. I've also had frame relay networks with MTU limitations, NICs with corrupted checksums, overflowing NAT tables, asymmetric ARP tables, misconfigured netmasks, and stuff I'm sure I've forgotten.
Re: TCP connection timeout mystery
#25are we ruling out content filtering? any content filter that is going to filter HTTPS without SSL decryption is going to look at the esni, which is in the client hello.
Re: TCP connection timeout mystery
#26Lots of good things to investigate already in the thread. I would throw in the potential for an anycast routing issue. TCP is stateful and if there is asymmetric routing, maybe the packets are coming from one anycast device, but the returning packets are routing to a different one.
Would suspect some of the other responses first though, but if they don't help this could be a possibility if they are using anycast.
Re: TCP connection timeout mystery
#27I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.
But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
Re: TCP connection timeout mystery
#2899% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.
Edit: on reading a few more comments, I think this is probably all wrong... The TLS Client hello is not that big (the client sent FIN is seq=518), and the server is only sending packets with SEQ=0. As others pointed out this likely means that the server that received the SYNs is not receiving the final ACK and data packets. From what I can tell, the example IP is not broadly anycast. From my test hosts in Seattle, tr…
ClientHello isn't that big but ServerHello that's in the reply can be quite large and since TCP packets have DF flag set, some middleware box may toss it if PMTUD didn't work correctly.
I had seen this exact issue with Fastly a few years ago.
Re: TCP connection timeout mystery
#29Earlier quoted context omitted.
Edit: on reading a few more comments, I think this is probably all wrong... The TLS Client hello is not that big (the client sent FIN is seq=518), and the server is only sending packets with SEQ=0. As others pointed out this likely means that the server that received the SYNs is not receiving the final ACK and data packets. From what I can tell, the example IP is not broadly anycast. From my test hosts in Seattle, tr…
ClientHello isn't that big but ServerHello that's in the reply can be quite large and since TCP packets have DF flag set, some middleware box may toss it if PMTUD didn't work correctly. I had seen this exact issue with Fastly a few years ago.
Yeah, I expected a large ServerHello, but then I would expect the server to send Seq=[LargeNumber] packets. Often you'd get an ACK for the ClientHello, then a missed packet or several, then the final packet of the ServerHello which is often small. Or at least an ack from the resend of ClientHello with a large sequence number.
I guess I've seen pmtud issues way too often in my life, and I just jumped ahead. :D
Re: TCP connection timeout mystery
#30I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.
But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.