I witnessed this issue as well in my previous workplace. I can't understand why Cross AZ is even a feature of NLB if this a problem.
Under what circumstances would anyone enable this? UDP only traffic?
AWS NLBs and the mixed up TCP connections
61–67 of 67 posts
Re: AWS NLBs and the mixed up TCP connections
#62Anyone have any context why the OPs post was removed from Reddit? https://www.reddit.com/r/aws/comments/jfx8af/aws_nlbs_and_th...
What makes you think it was removed? It's currently visible but nobody seems to have commented on it.
It's not visible unless you go directly to the post, which is how deleted posts work in Reddit. See:
https://snew.notabug.io/r/aws/comments/jfx8af/aws_nlbs_and_t...
Re: AWS NLBs and the mixed up TCP connections
#63Earlier quoted context omitted.
AFAIK, all the major tcp/ip stacks do round robin port assignment. Given relatively short, and relatively few, connections you should not have any collisions.
If your customers are behind CGNAT, and you have a good number of them, it would be pretty easy to run into collisions. CGNAT doesn't have any problem using the same source ip:port for connections to different destination ip:port, because there's no reason not to. I agree though, that most OS stacks are very conservative about using the same source ip:port for connections to anything else. It works ok until you need…
fwiw CGNAT is fairly uncommon in the United States except for cellphones. This creates one of those bad situations where you can have something that falls flat in the face of CGNAT but also "works fine" for all of your developer/employee traffic.
Re: AWS NLBs and the mixed up TCP connections
#64Earlier quoted context omitted.
What makes you think it was removed? It's currently visible but nobody seems to have commented on it.
It's not visible unless you go directly to the post, which is how deleted posts work in Reddit. See: https://snew.notabug.io/r/aws/comments/jfx8af/aws_nlbs_and_t...
Ah, the one that's been on the page all day is a cross-post from r/devops:
https://www.reddit.com/r/aws/comments/jg5k7f/aws_nlbs_and_th...
Re: AWS NLBs and the mixed up TCP connections
#65If one is using proxy protocol within your stack, it would seem like this could be a good work-around here. Particularly if having knowledge of the original client source IP is important to your application.
Has anyone tried this?
Re: AWS NLBs and the mixed up TCP connections
#66Earlier quoted context omitted.
It still data center. And when you choose the low latency instance placement means that all your virtual machines are placed in the same rack and/or host. Basically throwing availability over the window.
>And when you choose the low latency instance placement means that all your virtual machines are placed in the same rack and/or host That's also not true.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... check it out :)
Re: AWS NLBs and the mixed up TCP connections
#67Earlier quoted context omitted.
There are two modes to NLB: 'instance' mode which is what they are describing in the article. I think what happens is the packets hit the AWS network and then somehow AWS decides whether it is an existing flow or chooses a target instance to create a new flow. then it just routes the packet by only modifying the destination address. it can do this because the whole of the AWS network is basically a lie and ip packets…
Ok, ip mode seems usable if you don't have a lot of connections per NLB/instance pair, but kind of meh. Does the instance port have to match the service port? In instance mode, you could setup so us-east1 is like 441, us-east2 is 442, us-west1 is 451 etc. And then you wouldn't have tcp 4-tuple collisions on the instance. Similarly for ip mode, you could put the same host in for ports 440-449 and get 10x 55,000 connec…
That AZ-port solution would work as well and would be easier to configure than different IPs. But they don't let you map port or IP based on AZ for a single NLB. You could setup different independent NLBs in each zone and then setup target groups for each NLB AZ and I think this might have been what you were proposing. I'm pretty sure you can setup a different port for each target group and it doesn't have to match the traffic port on the NLB.
For IP mode you are correct that you can run on multiple ports to get around the connection limit. The documentation even says it is based on IP address and port.
> When the target type is ip, the load balancer can support 55,000 simultaneous connections or about 55,000 connections per minute to each unique target (IP address and port). If you exceed these connections, there is an increased chance of port allocation errors. If you get port allocation errors, add more targets to the target group.
https://docs.aws.amazon.com/elasticloadbalancing/latest/netw...