Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TCP connection timeout mystery

devnonsense.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: TCP connection timeout mystery

#31

Post by Existing4190 »

js2 wrote: post #27
Existing4190 wrote: post #18

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

You are right. My comment can't solve the whole story.

Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

Re: TCP connection timeout mystery

#32

Post by axus »

Thev00d00 wrote: post #22
Existing4190 wrote: post #18

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

Yeah, this was my immediate thought, someone has made the 169.254 block too large somewhere.

They should go to http://169.150.211.2 and see if that gets blocked. I get a "Welcome to nginx!" page there.

Re: TCP connection timeout mystery

#33

Post by Izmaki »

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS.

Like...

"My car won't start."

"Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

Re: TCP connection timeout mystery

#34

Post by bbss »

outsidein wrote: post #7

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

Agreed, my best guess it's due to a smaller MTU between the CDN and your device. They are probably replying with TLS Server Hello which would typically max a standard 1500 byte packet. It's also likely why HTTP isn't working either since they would ACK the connection, you would probably be able to issue the GET / but you would never get a response back due to the HTTP response payload being larger than a single packet.

A few ideas to test this theory: 1) Find an asset on their server that is smaller than 500-1000 bytes so the entire payload will fit in a packet. Maybe a HEAD would work? 2) Clamp your MSS on this IP to something much smaller like 500 instead of the standard 1460. This should force the server to send smaller packets and will work better in practice than changing your MTU. See: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.cookbook.mtu-...

Re: TCP connection timeout mystery

#35

Post by gjf »

First off, the HTTP HTTP 301s to the HTTPS site, so HTTPS is still the likely trigger.

Second, I see that whatever client he's using is specifying a very old TLS 1.0. If its not MTU (which others have mentioned), then my guess would be a firewall with a policy specifying a minimum TLS version, and dropping this connection on the floor.

Re: TCP connection timeout mystery

#36

Post by dark-star »

Wow. What an embarassing answer by the "Berkeley Student Tech Services"...

That is on the same level as e.g. the customer hotline at a phone company ("did you try turning it off and on again?"), I would have thought that Berkeley of all university has higher standards than that

Re: TCP connection timeout mystery

#37

Post by iforgotpassword »

dark-star wrote: post #36

Wow. What an embarassing answer by the "Berkeley Student Tech Services"... That is on the same level as e.g. the customer hotline at a phone company ("did you try turning it off and on again?"), I would have thought that Berkeley of all university has higher standards than that

It's not like they know anything about the Internet there or ever created anything for it that's still in use... ;)

It's indeed sad how more and more unis outsource all their IT. Like they've become too stupid to manage the tech they created. A friend of mine just told me how his old college is currently moving their email to Google, and are also looking to move all the web hosting somewhere else. What's next, have the whole network managed by Comcast? Pay per connected device?

Re: TCP connection timeout mystery

#38

Post by kjs3 »

oasisbob wrote: post #19

Feels like some stateful device within someone's network mishanding the connection state, like the author guesses. It's interesting that your side thinks the three-way handshake worked, but the remote side continues to resend the [SYN, ACK] packets, as if they've never received the final [ACK] from you. Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers - there was a bug in…

Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers

We diagnosed the same(ish) bug in first generation F5 LBs in the 90s[1]. Figured exhaustive testing for this would have been SOP by now.

[1] To be fair, almost all 1st gen LBs had at least one major "send the packet to the wrong place and the state table gets screwed up" bug.

Re: TCP connection timeout mystery

#39

Post by Asaf51 »

outsidein wrote: post #7

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

The Ack in TCP handshake is obviously dropped (as the server resends SYN+ACK). So probably has nothing related to MTU.

Re: TCP connection timeout mystery

#40

Post by [deleted] »

gjf wrote: post #35

First off, the HTTP HTTP 301s to the HTTPS site, so HTTPS is still the likely trigger. Second, I see that whatever client he's using is specifying a very old TLS 1.0. If its not MTU (which others have mentioned), then my guess would be a firewall with a policy specifying a minimum TLS version, and dropping this connection on the floor.

[deleted]
Post reply on HN