Live data from Hacker News

TCP connection timeout mystery

devnonsense.com

61–70 of 83 posts

Re: TCP connection timeout mystery

#61

I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.

My employer did something like that once, and it took out access to github.

Re: TCP connection timeout mystery

#62

Earlier quoted context omitted.

> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. Are you referring to local networks only? It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.

That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.

AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).

Re: TCP connection timeout mystery

#63
post #53
post #30

Earlier quoted context omitted.

Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.

That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.

It's been a very long time since I've diagnosed something like this, but I've had problems in the past when the MTU is smaller than the default and ICMP is blocked (interfering with path MTU discovery). Often IPSec or some other tunneling was involved. The initial packets got through but as soon as a full packet was sent it was dropped.

EDIT - I've now scrolled down in HN and saw that this was ruled out.

Re: TCP connection timeout mystery

#64

Earlier quoted context omitted.

That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.

AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).

This explains why some people have problems with IPv6 - if you block IPv6 Control Messages, then it will only work sometimes.

Re: TCP connection timeout mystery

#65
post #27

Earlier quoted context omitted.

But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.

You are right. My comment can't solve the whole story. Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.

Filtering + established connection tracking:

https://news.ycombinator.com/item?id=39822214

Re: TCP connection timeout mystery

#66
Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic.

In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with seq=0 ack=1, repeating a few times later, same as OP.

So as others suggested, likely a misconfigured BOGON rule with 169.0.0.0/8, but also matching outbound established connections rather than new/any state for some reason.

[0] https://i.imgur.com/AwJGI3W.png

Re: TCP connection timeout mystery

#67
post #54
post #44

Earlier quoted context omitted.

Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.

[deleted]

Similar happened to me not so long ago. One day a junior admin asked me to diagnose who ssh to a box started hanging. After a bit of diagnosis it became evident TCP's 3 way handshake got through, and then it all stopped. No normal network behaves like this.

My answer was "some network admin is having fun with a middleware box", you will have to speak with them. They did, and the response that came back is "we are moving to a network with real security". Access was restored. It was a Palo Alto NGFW as others have mentioned.

IMHO the industry looking for way to move high priced gear, and they convince someone with a sparrow problem with barely knows how to handle an air rifle to buy bazooka's. A bit of collateral damage is to be expected....

Re: TCP connection timeout mystery

#68

Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic. In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with…

Good find, that fits the symptoms perfectly and is more likely than not a problem with the firewall on the source end (the campus network). Did you email the author?

Re: TCP connection timeout mystery

#70
post #68

Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic. In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with…

Good find, that fits the symptoms perfectly and is more likely than not a problem with the firewall on the source end (the campus network). Did you email the author?

As a network engineer it piqued my interest (unemployment is booring) as there were no completely satisfying answers, though some were close. Thought it was the old MTU problem at first but as it was the ack of the handshake being retransmitted it wasn't likely. So tried a few things with my router.

This is how you get NOCs to help you quickly, give them not only the problem but the root cause as well. Its not that they (or me) are lazy, its just that it can be so many things that can be a potential cause of problems, especially when you only have incomplete information to go on.

Post reply on HN