Live data from Hacker News

TCP connection timeout mystery

devnonsense.com

11–20 of 83 posts

Re: TCP connection timeout mystery

#11
My guess would be something related to your campus having more than one external connection available.

Maybe from the server's point of view the SYN and ACK are coming from distinct addresses and this is tripping them up ?

I have 2 internet connection in my home and would encounter some strange bugs whenever I used both connections at the same time. I never debbuged theses cases but they always disappeared when I just used 1 connection and left the second as a backup.

Re: TCP connection timeout mystery

#12

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

The article mentions that it happens both over HTTP and HTTPS.

I'd ask OP to check if this is only affects a subset of their IPs from https://bunnycdn.com/api/system/edgeserverlist, or whether all of their IPs are affected using `curl --resolve bunnycdn-hosted-website.com:80:some-other-ip http://bunnycdn-hosted-website.com`.

Re: TCP connection timeout mystery

#13
I'm rather surprised that Berkeley Student Tech Services would keep people around who either don't know how DNS works or know, but who make up excuses to dismiss a problem.

The problem really should be escalated and the nonsense answer pointed out, because if they care (and they should), they'll want to educate the person who gave that response.

Re: TCP connection timeout mystery

#15

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

I believe this is relatively easy to test as I think you can gradually increase the size of the ICMP packet until it stops responding. I have done something along those lines in the past but it was a long time ago.

Re: TCP connection timeout mystery

#16

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

The article mentions that it happens both over HTTP and HTTPS. I'd ask OP to check if this is only affects a subset of their IPs from https://bunnycdn.com/api/system/edgeserverlist , or whether all of their IPs are affected using `curl --resolve bunnycdn-hosted-website.com:80:some-other-ip http://bunnycdn-hosted-website.com `.

Besides that, the author points out that the final handshake ACK never reaches the server and that packet is small, not going to go over the mtu.

Re: TCP connection timeout mystery

#17
post #14

Maybe it’s not a network issue at all - might be related to a purposeful action taken by a network device (ips or web filter etc) that is killing the connection based on some rule set.

It's possible but the way the connection is blocked is surprising. If you're blocking based on an IP you'd just drop the first syn and the client would never receive the syn-ack. If you're blocking based on the SNI you would be waiting for the first TLS client-hello, but in that case packet are droped before the client-hello is sent.

Re: TCP connection timeout mystery

#18
I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall

I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

Re: TCP connection timeout mystery

#19
Feels like some stateful device within someone's network mishanding the connection state, like the author guesses.

It's interesting that your side thinks the three-way handshake worked, but the remote side continues to resend the [SYN, ACK] packets, as if they've never received the final [ACK] from you.

Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers - there was a bug in the hashing implementation used to assign TCP flows to different CPUs. If you hit this bug (parts per thousand), your connection would be assigned to a CPU with no record of that flow existing, so the connection would be alive, but would no longer pass packets. Would take a long time for the local TCP stack to go through its exponential retries and finally decide to drop the connection and start over .

Re: TCP connection timeout mystery

#20
I wouldn't be surprised if someone (your Uni) is mistakenly blocking some 169.x.x.x data since 169.254.0.0/16 is used for local IPs. Someone put the wrong subnet mask in a firewall rule or ACL someplace.
Post reply on HN