Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TCP connection timeout mystery

devnonsense.com

Post reply on HN
Print view View original

41–50 of 83 posts

Re: TCP connection timeout mystery

#41

Post by throwway120385 »

Izmaki wrote: post #33

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

And like what does DNS have to do with packets being dropped? The name is already resolved to an IP address at this point and we're seeing a SYN and SYN+ACK, which tells me that it's not a routing issue. The fact that it happens at the start of a TLS connection(Client Hello) makes me think that it's some kind of web application firewall or reverse proxy or some other intervening firewall that's causing this.

Re: TCP connection timeout mystery

#42

Post by vadiml »

krypd0h wrote: post #20

I wouldn't be surprised if someone (your Uni) is mistakenly blocking some 169.x.x.x data since 169.254.0.0/16 is used for local IPs. Someone put the wrong subnet mask in a firewall rule or ACL someplace.

This would prevent initial SYN To reach the server too.

Re: TCP connection timeout mystery

#43

Post by xnyan »

js2 wrote: post #27
Existing4190 wrote: post #18

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

Re: TCP connection timeout mystery

#44

Post by xnyan »

Existing4190 wrote: post #31
js2 wrote: post #27

Earlier quoted context omitted.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

You are right. My comment can't solve the whole story. Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.

Re: TCP connection timeout mystery

#45

Post by deeviant »

Somewhat related anecdote:

Some 10 years back I was working for a solar company doing SCADA stuff (monitoring remote power plant equipment, reporting generation metrics, handling grid interconnect stuff, etc).

We had a big room with lots of monitors that looked like a set in a Hollywood film, no doubt inspired by them. You could see all the solar installations all around the world that we monitored. The monitoring crew put out a call for engineers, stat, and as I walked into the monitoring room I could see perhaps 1/10th of the power plant icons on the wall we red "lost communication", one plant went green to red right in front of me.

This started a shitstorm with all hands be summoned. Long story short, somebody decided the best way to get an external IP for one of our remote gateways was to use curl command to a whatismyip.com type service, but instead of targeting Google (or you know, a server under our control), it hit some random ISP in Italy. The ISP most have eventually realized they were getting ping on by thousands of devices 24/7, so they decided they would drop some percentage of incoming requests silently, and of course the curl call was blocking without timeout. When the remote gateway's was dropped, it blocked indefinitely.

I skipped a lot in between but it was definitely a fun firefighting session, it was particularly hampered by a couple engineers that were quite high up on the food chain getting lead in the wrong direction (as to the root cause) at the beginning and fighting particularly hard against any opposing theories. It was the one time I basically got to drop the "I'm right and I bet my job on it." Fun times.

Re: TCP connection timeout mystery

#47

Post by whirlwin »

xnyan wrote: post #43
js2 wrote: post #27

Earlier quoted context omitted.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases.

Are you referring to local networks only?

It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.

Re: TCP connection timeout mystery

#48

Post by fragmede »

whirlwin wrote: post #47
xnyan wrote: post #43

Earlier quoted context omitted.

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. Are you referring to local networks only? It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.

That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.

Re: TCP connection timeout mystery

#49

Post by johnp_ »

gjf wrote: post #35

First off, the HTTP HTTP 301s to the HTTPS site, so HTTPS is still the likely trigger. Second, I see that whatever client he's using is specifying a very old TLS 1.0. If its not MTU (which others have mentioned), then my guess would be a firewall with a policy specifying a minimum TLS version, and dropping this connection on the floor.

Certainly weird that wireshark shows TLSv1 while curl shows TLSv1.3. That shouldn't happen unless something interfered with the Client Hello. (or the wireshark version is outdated)
Post reply on HN