Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TCP connection timeout mystery

devnonsense.com

Post reply on HN
Print view View original

51–60 of 83 posts

Re: TCP connection timeout mystery

#51

Post by gregw2 »

johnp_ wrote: post #49
gjf wrote: post #35

First off, the HTTP HTTP 301s to the HTTPS site, so HTTPS is still the likely trigger. Second, I see that whatever client he's using is specifying a very old TLS 1.0. If its not MTU (which others have mentioned), then my guess would be a firewall with a policy specifying a minimum TLS version, and dropping this connection on the floor.

Certainly weird that wireshark shows TLSv1 while curl shows TLSv1.3. That shouldn't happen unless something interfered with the Client Hello. (or the wireshark version is outdated)

Ran into this myself about 10 days ago.

If a TLS handshake is aborted partway through, Wireshark will label it “TLSv1”. It actually retroactively labels the 1.0 TLS packets as 1.3 after a successful TLS 1.3 handshake finishes.

This makes sense because a TLSv1.3 handshake actually starts as 1.0 and then upgrades to 1.3 only with IIRC the Server Hello response to the ClientHello.

The following links document this behavior, in case you or your organization’s security team is nervous TLSv1 is actually being used:

https://superuser.com/a/1618420

https://ask.wireshark.org/question/24276/how-does-wireshark-...

https://gitlab.com/wireshark/wireshark/-/issues/16114

Re: TCP connection timeout mystery

#52

Post by js2 »

xnyan wrote: post #43
js2 wrote: post #27

Earlier quoted context omitted.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

ICMP (the protocol ping uses) is a totally separate protocol from TCP and UDP. Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. BTW your assumption "a successful ICMP ping = TCP and UDP work" is an extremely common one that I too had before I was taught otherwise.

> BTW your assumption

I did not assume. The comment to which I was responding suggested it was the destination IP that was the problem. Generally (but not always) an IP filter would be applied irrespective of protocol. I also pointed out that the initial SYN and reply SYN/ACK are getting through the hypothesized bogon filter and those are part of TCP. I don't think the bogon filter is a hypothesis that fits the evidence.

ETA: but adding connection state tracking + a filter does make sense https://news.ycombinator.com/item?id=39822214

Re: TCP connection timeout mystery

#53

Post by js2 »

xyst wrote: post #30
js2 wrote: post #27

Earlier quoted context omitted.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.

That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.

Re: TCP connection timeout mystery

#54

Post by [deleted] »

xnyan wrote: post #44
Existing4190 wrote: post #31

Earlier quoted context omitted.

You are right. My comment can't solve the whole story. Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.

[deleted]

Re: TCP connection timeout mystery

#55

Post by nijave »

Izmaki wrote: post #33

What really grinds my gears is that a networking team believes the culprit is a static DNS that "conflicts" with their DNS. Like... "My car won't start." "Oh, OK, have you tried waiting for the traffic lights to go green, as designed by the Principal Road Engineer?"

My guess is it either got some boilerplate response from L2 instead of actually going to a network engineer or it did go to a network engineer but they're connecting from a different network with different traffic management and don't see the issue.

At my old uni, L1 were paid students, L2 were paid staff, and L3 were the actual netops/sysadmins so sometimes L2 would try to close something out that needed escalated.

In addition, they had resnet (residential network) and pronet (professional network) where the former was for student housing and the latter everything else. Resnet had more restrictions and traffic shaping such that pronet traffic was prioritized. In addition, resnet wireless had a different NAT setup whereas resnet wired used public IPs with inbound traffic blocked. This lead to all kinds of caveats like online gaming using uPnP only working on wireless despite wired having public IPs.

Re: TCP connection timeout mystery

#56

Post by vlan0 »

johnklos wrote: post #13

I'm rather surprised that Berkeley Student Tech Services would keep people around who either don't know how DNS works or know, but who make up excuses to dismiss a problem. The problem really should be escalated and the nonsense answer pointed out, because if they care (and they should), they'll want to educate the person who gave that response.

You’d think that. But having spent time higher for operations/support in higher ed, it’s really hard to attract people that have a quality foundation of knowledge.

We don’t pay enough

Re: TCP connection timeout mystery

#58

Post by toast0 »

nathanyz wrote: post #26

Lots of good things to investigate already in the thread. I would throw in the potential for an anycast routing issue. TCP is stateful and if there is asymmetric routing, maybe the packets are coming from one anycast device, but the returning packets are routing to a different one. Would suspect some of the other responses first though, but if they don't help this could be a possibility if they are using anycast.

I don't think the IP shared is anycast. All of my personal test nodes are Seattle based, and they all see the same basic path to the IP that was shared; transit to San Jose, then two hops in BunnyCDN's network. Additionally, I get a different IP when I lookup the test hostname, that traces to Seattle.

It does feel like maybe a different server/network path getting the SYN+ACK vs the ACK, but probably in BunnyCDN's equipment --- but maybe something weird in Berkeley's (wired) network causes weird behavior for BunnyCDN? Hard to really know without pcaps from both ends, which are hard to get. Something funky in the load balancer seems like a good guess to me.

Re: TCP connection timeout mystery

#59

Post by johnp_ »

gregw2 wrote: post #51
johnp_ wrote: post #49

Earlier quoted context omitted.

Certainly weird that wireshark shows TLSv1 while curl shows TLSv1.3. That shouldn't happen unless something interfered with the Client Hello. (or the wireshark version is outdated)

Ran into this myself about 10 days ago. If a TLS handshake is aborted partway through, Wireshark will label it “TLSv1”. It actually retroactively labels the 1.0 TLS packets as 1.3 after a successful TLS 1.3 handshake finishes. This makes sense because a TLSv1.3 handshake actually starts as 1.0 and then upgrades to 1.3 only with IIRC the Server Hello response to the ClientHello. The following links document this behav…

Oh, indeed, that's quite surprising. A TLSv1.3 Client Hello always contains the supported_versions extension, which should allow wireshark to label it correctly, regardless of whether or not the handshake actually finishes. Though, tbf, it does say TLSv1 and not TLSv1.0. I wonder how it would look had TLSv1.3 been named TLSv2.0 after all...

edit: Ah, that GitLab link lead me to https://gitlab.com/wireshark/wireshark/-/issues/19515 which is a recent discussion around this topic and https://gitlab.com/wireshark/wireshark/-/merge_requests/1377... already dealt with it :)

Re: TCP connection timeout mystery

#60

Post by Hikikomori »

supriyo-biswas wrote: post #12
outsidein wrote: post #7

99% MTU size. Had this recently specifically with TLS due to large initial packets containing certificates. Results could even depend on user agent, some fail some will work. try to reduce MTU on client, 1280 is a good starting point.

The article mentions that it happens both over HTTP and HTTPS. I'd ask OP to check if this is only affects a subset of their IPs from https://bunnycdn.com/api/system/edgeserverlist , or whether all of their IPs are affected using `curl --resolve bunnycdn-hosted-website.com:80:some-other-ip http://bunnycdn-hosted-website.com `.

But we don't know the full story of http as no capture was provided. Typically when you have an mtu issue you would get stuck on the tls handshake, as we are in this case for Https, so in the http capture we should see a 301 redirect if it's an mtu issue.
Post reply on HN