Live data from Hacker News

TCP connection timeout mystery

devnonsense.com

71–80 of 83 posts

Re: TCP connection timeout mystery

#71
post #33

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

At least they're consistent with the "it's always DNS" meme.

Re: TCP connection timeout mystery

#73

Feels like some stateful device within someone's network mishanding the connection state, like the author guesses. It's interesting that your side thinks the three-way handshake worked, but the remote side continues to resend the [SYN, ACK] packets, as if they've never received the final [ACK] from you. Had a hellish time troubleshooting a similar problem several years ago with F5 load balancers - there was a bug in…

Bogon filter for 169/8 that matches established connections on outbound.

Re: TCP connection timeout mystery

#74
post #42
post #20

I wouldn't be surprised if someone (your Uni) is mistakenly blocking some 169.x.x.x data since 169.254.0.0/16 is used for local IPs. Someone put the wrong subnet mask in a firewall rule or ACL someplace.

This would prevent initial SYN To reach the server too.

Not if you also match on established connections and outbound only.

Re: TCP connection timeout mystery

#75

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

It's the ack in the handshake that is dropped. A filter for 169/8 matching on established connections and only outbound would cause that.

Re: TCP connection timeout mystery

#76
post #8
post #5

Earlier quoted context omitted.

I mean, yes, that was my instinctive response based on just the title. It's always the MTU. But in this case the packet that's being lost is a pure ACK.

Looks to me like the TLS client hello is being lost, which is why the server is sending duplicate SYN+ACKs.

No. To repeat: the client is sending a pure ACK with no data. You can see that packet with a 0 byte payload length in the screenshot. That packet is also getting lost; if it weren't getting lost, we'd be receiving a SYNACK.

Re: TCP connection timeout mystery

#77

Somewhat related anecdote: Some 10 years back I was working for a solar company doing SCADA stuff (monitoring remote power plant equipment, reporting generation metrics, handling grid interconnect stuff, etc). We had a big room with lots of monitors that looked like a set in a Hollywood film, no doubt inspired by them. You could see all the solar installations all around the world that we monitored. The monitoring cr…

Do you get double your salary if you win? ;-P

Re: TCP connection timeout mystery

#78
post #55
post #33

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

My guess is it either got some boilerplate response from L2 instead of actually going to a network engineer or it did go to a network engineer but they're connecting from a different network with different traffic management and don't see the issue. At my old uni, L1 were paid students, L2 were paid staff, and L3 were the actual netops/sysadmins so sometimes L2 would try to close something out that needed escalated.…

Regardless of which network they connect from, I would expect that a network engineer knows that if a TCP handshake with the web server (i.e. after DNS lookup) fails at the 3rd step, then it's not DNS. The fact that the TCP handshake begun is evidence that DNS works.

Re: TCP connection timeout mystery

#79
post #33

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

They just wanted to close the ticket.

All that explanation is just ritual -- it does not need to make sense.

Re: TCP connection timeout mystery

#80
post #52
post #43

Earlier quoted context omitted.

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

> BTW your assumption I did not assume. The comment to which I was responding suggested it was the destination IP that was the problem. Generally (but not always) an IP filter would be applied irrespective of protocol. I also pointed out that the initial SYN and reply SYN/ACK are getting through the hypothesized bogon filter and those are part of TCP. I don't think the bogon filter is a hypothesis that fits the evide…

> Generally (but not always) an IP filter would be applied irrespective of protocol

I've never seen this (all protocols by default) in any environment I've personally worked with, but perhaps I've had a unique experience.

Post reply on HN