I'm suspicious about the IP 169.150.221.147 My guess: there is some misconfigured bogons IP filter and instead of 169.254.0.0/16 (rfc3927) there is something like 169.0.0.0/8 configured to be blocked on some firewall I once was a customer of an ISP that mistakenly blocked the whole 192.0.0.0/8 net, which caused some confusion, but they fixed it after I pointed it out.
TCP connection timeout mystery
61–70 of 83 posts
Re: TCP connection timeout mystery
#62Earlier quoted context omitted.
> Blocking ICMP can break of lot of things and offers no real benefits outside of a handful of specific edge cases. Are you referring to local networks only? It's very common to not allow ICMP by defaul to workloads in the cloud, e.g. in AWS.
That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.
Re: TCP connection timeout mystery
#63Earlier quoted context omitted.
Probably because the firewall rule only includes TCP/UDP. ICMP is often not blocked, in my experience.
That doesn't explain the initial SYN/ACK and reply ACK which are part of the TCP establishment and that we see getting through.
EDIT - I've now scrolled down in HN and saw that this was ruled out.
Re: TCP connection timeout mystery
#64Earlier quoted context omitted.
That's likely to be an implementation detail of how they've implemented TCP routing across a large fabric.
AWS doesn't decide or even care about this, customers configure security group rules for their own services. Nothing is allowed by default, so if you want ICMP you would need to allow it, most font bother because it's not that helpful in a cloud environment (can just monitor the TCP port instead and get similar information).
Re: TCP connection timeout mystery
#65Earlier quoted context omitted.
But then why would the ICMP echo/reply (ping) be allowed through? And how is the initial syn/ack and reply getting through? It's only the second ack that's getting (apparently) blocked.
You are right. My comment can't solve the whole story. Still, some middlebox/stateful firewall/etc. messing with 169.0.0.0/8 is plausible.
Re: TCP connection timeout mystery
#66In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with seq=0 ack=1, repeating a few times later, same as OP.
So as others suggested, likely a misconfigured BOGON rule with 169.0.0.0/8, but also matching outbound established connections rather than new/any state for some reason.
Re: TCP connection timeout mystery
#67Earlier quoted context omitted.
Actually I think you might still be right. Ping uses ICMP, which is almost never blocked in my experience. I learned that because early in my career I too assumed a successful ping = TCP and UDP also work.
[deleted]
My answer was "some network admin is having fun with a middleware box", you will have to speak with them. They did, and the response that came back is "we are moving to a network with real security". Access was restored. It was a Palo Alto NGFW as others have mentioned.
IMHO the industry looking for way to move high priced gear, and they convince someone with a sparrow problem with barely knows how to handle an air rifle to buy bazooka's. A bit of collateral damage is to be expected....
Re: TCP connection timeout mystery
#68Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic. In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with…
Re: TCP connection timeout mystery
#69Re: TCP connection timeout mystery
#70Think I was able to reproduce it. I configured my router to drop established connections for IP 169.150.221.147 in my policy attached to my wan interface for outgoing traffic (important detail, inbound would drop the syn/ack instead). For reference its an Ubiquiti Edgerouter that uses iptables to filter traffic. In the linked picture [0] I have packet #436 selected, its a retransmission of the handshake syn/ack with…
Good find, that fits the symptoms perfectly and is more likely than not a problem with the firewall on the source end (the campus network). Did you email the author?
This is how you get NOCs to help you quickly, give them not only the problem but the root cause as well. Its not that they (or me) are lazy, its just that it can be so many things that can be a potential cause of problems, especially when you only have incomplete information to go on.