31–40 of 67 posts
Anyone have any context why the OPs post was removed from Reddit? https://www.reddit.com/r/aws/comments/jfx8af/aws_nlbs_and_th...
Would incidents of this be captured by the "node_netstat_TcpExt_TCPAbortOnData" metric ("connections reset due to unexpected data")?
This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ! If you need to call out of the the AZ for other data or API sources, either figure o…
Does anyone know a good way to observe & measure the impact of this? I have a small fleet of Linux servers behind NLBs with cross-AZ load balancing turned on, and they have some significant collection of OS-level metrics via the Prometheus node_exporter. Would incidents of this be captured by the "node_netstat_TcpExt_TCPAbortOnData" metric ("connections reset due to unexpected data")?
It annoys me to no end when people don't explain their abbreviations... "AZ" means Availability Zone, which is to say: Data Center. So cross-AZ means going to multiple data centers.
An availability zone isn't equivalent to a data center, as it might consist of multiple data centers. A better explanation for availability zone would be "a bunch of data centers in close physical proximity, exposed to users as a single logical entity". Or as AWS explains it [1]: > An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZ’s…
And when you choose the low latency instance placement means that all your virtual machines are placed in the same rack and/or host.
Basically throwing availability over the window.
Earlier quoted context omitted.
An availability zone isn't equivalent to a data center, as it might consist of multiple data centers. A better explanation for availability zone would be "a bunch of data centers in close physical proximity, exposed to users as a single logical entity". Or as AWS explains it [1]: > An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZ’s…
It still data center. And when you choose the low latency instance placement means that all your virtual machines are placed in the same rack and/or host. Basically throwing availability over the window.
That's also not true.
they could fix the problem by assigning each NLB AZ IP an aliased IP on your target machine. however, this would be an absolute mess to configure. For example if the 3 NLB AZ IPs were: 1.2.3.6 1.2.3.7 1.2.3.8 then if you set up 3 IP aliases on your target instance: 192.168.2.6 192.168.2.7 192.168.2.8 then if the NLB always mapped: 1.2.3.6 -> 192.168.2.6 1.2.3.7 -> 192.168.2.7 1.2.3.8 -> 192.168.2.8 then there would b…
It seems like it's basically doing NAT to take client:port server:port to client:port instance:port, including adjusting instance NAT mappings so the outgoing traffic gets NATed back?
Sort of like direct server return load balancing, but weird because the instance doesnt see the service IP and NAT takes care of it, somehow?
In that case, yeah, you either need to have a rfc1918 service address mapped for each public service address or just use the public address.
Elsewhere in thread, it sounds like it's possible to set this up using the public address but kubernetes doesn't like that; in which case, it seems appropriate to fix kubernetes.
Earlier quoted context omitted.
Collisions can already occur at two connections
AFAIK, all the major tcp/ip stacks do round robin port assignment. Given relatively short, and relatively few, connections you should not have any collisions.
I agree though, that most OS stacks are very conservative about using the same source ip:port for connections to anything else. It works ok until you need to make lots of connections, then you have to manage ports in userspace.
How does an NLB handle the following two connections:
1) from 1.2.3.4:44444 to NLB:443 2) from 1.2.3.5:44444 to NLB:443
What does the (single) inside server see?
they could fix the problem by assigning each NLB AZ IP an aliased IP on your target machine. however, this would be an absolute mess to configure. For example if the 3 NLB AZ IPs were: 1.2.3.6 1.2.3.7 1.2.3.8 then if you set up 3 IP aliases on your target instance: 192.168.2.6 192.168.2.7 192.168.2.8 then if the NLB always mapped: 1.2.3.6 -> 192.168.2.6 1.2.3.7 -> 192.168.2.7 1.2.3.8 -> 192.168.2.8 then there would b…
Since you seem knowledgable and I'm not an AWS customer... Can you confirm the general workings of the Network Load Balancer? It seems like it's basically doing NAT to take client:port server:port to client:port instance:port, including adjusting instance NAT mappings so the outgoing traffic gets NATed back? Sort of like direct server return load balancing, but weird because the instance doesnt see the service IP and…
'instance' mode which is what they are describing in the article. I think what happens is the packets hit the AWS network and then somehow AWS decides whether it is an existing flow or chooses a target instance to create a new flow. then it just routes the packet by only modifying the destination address. it can do this because the whole of the AWS network is basically a lie and ip packets can just magically pop up in your interface and they don't have to go through a normal routing process. then on the way back it is able to track the flow of the connection and know that has to rewrite the source address of the packet. i think this is basically what you describe above.
then there is 'ip' mode which i think it looks like it works similar to how haproxy would proxy TCP traffic. all of the packets you receive look like they come from the NLB and not the client. because there can be many clients this means the NLB networking layer needs to rewrite the source address and the source port which i guess is a more traditional kind of NAT. this also means there is a limit to the number of concurrent flows you can have. in the documentation i think AWS says there is 55,000 connection limit.