Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS NLBs and the mixed up TCP connections

niels-ole.com

Post reply on HN
Print view View original

21–30 of 67 posts

Re: AWS NLBs and the mixed up TCP connections

#21

Post by Dunedan »

kerblang wrote: post #18

It annoys me to no end when people don't explain their abbreviations... "AZ" means Availability Zone, which is to say: Data Center. So cross-AZ means going to multiple data centers.

An availability zone isn't equivalent to a data center, as it might consist of multiple data centers. A better explanation for availability zone would be "a bunch of data centers in close physical proximity, exposed to users as a single logical entity".

Or as AWS explains it [1]:

> An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZ’s give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and scalable than would be possible from a single data center.

[1]: https://aws.amazon.com/about-aws/global-infrastructure/regio...

Re: AWS NLBs and the mixed up TCP connections

#22

Post by nielsole »

beowulfe wrote: post #13

I debugged and identified the exact same problem a few weeks ago. I don't have any solutions, but can confirm what you're seeing. I suspect most clients aren't creating enough tcp connections in the window to cause a collision. (In our case, we discovered the issue from our load box during performance testing)

Collisions can already occur at two connections

Re: AWS NLBs and the mixed up TCP connections

#23

Post by wjossey »

base wrote: post #16

If you have cross zone load balancing disabled and you only have one instance per AZ, how do you ensure that when an instance is down or during a deploy, when not all AZ might have instances, that you don't have downtimes? Cross zone load balancing in NLB seems a must to me if you are constantly deploying new targets.

There’s a few strategies you can employ. One is to roll each AZ one at a time, and take each out of service first.

Another strategy is to have redundancy in each AZ such that your rolling strategy never takes the AZ itself offline.

Just depends on how much infra you want to manage to be honest. Taking an AZ offline can be as easy as having an ingress hop that signals back to the LBs that it is now failing health checks and should be temporarily removed from service.

Re: AWS NLBs and the mixed up TCP connections

#24

Post by nielsole »

dopylitty wrote: post #17
rocqua wrote: post #6

What would be required for the NLB host to do port-rewriting to make things stable? Is the NLB host not present on the return-path? If that is not the issue, is this a performance issue? Could random allocation of ports work instead?

My guess is the network routing is happening in the software defined network AWS creates below the traditional networking layers. Sometimes it's tempting to think AWS' services are just like traditional appliances you'd stick in the middle of a network path but in reality they're all virtual and run on top of their software defined network in something they call hyperplane. That means traditional intuition about how…

> they need to preserve the static public IP that was used on the NLB for the incoming request

Even if they did that, to determine which outgoing public IP the packet should go to, they would need to guess based on segment numbers and ack offsets. This would still be a guessing game

Re: AWS NLBs and the mixed up TCP connections

#25

Post by beowulfe »

nielsole wrote: post #22
beowulfe wrote: post #13

I debugged and identified the exact same problem a few weeks ago. I don't have any solutions, but can confirm what you're seeing. I suspect most clients aren't creating enough tcp connections in the window to cause a collision. (In our case, we discovered the issue from our load box during performance testing)

Collisions can already occur at two connections

AFAIK, all the major tcp/ip stacks do round robin port assignment. Given relatively short, and relatively few, connections you should not have any collisions.

Re: AWS NLBs and the mixed up TCP connections

#26

Post by dopylitty »

nielsole wrote: post #24
dopylitty wrote: post #17

Earlier quoted context omitted.

My guess is the network routing is happening in the software defined network AWS creates below the traditional networking layers. Sometimes it's tempting to think AWS' services are just like traditional appliances you'd stick in the middle of a network path but in reality they're all virtual and run on top of their software defined network in something they call hyperplane. That means traditional intuition about how…

> they need to preserve the static public IP that was used on the NLB for the incoming request Even if they did that, to determine which outgoing public IP the packet should go to, they would need to guess based on segment numbers and ack offsets. This would still be a guessing game

Oh for sure. Distributed routing like this is complex!

My main point is we have no idea what AWS' SDN looks like under the covers. Traditional rules about routing may not apply and each node in the path may have a lot more information about traffic than a traditional router would have.

Re: AWS NLBs and the mixed up TCP connections

#27

Post by nielsole »

beowulfe wrote: post #25
nielsole wrote: post #22

Earlier quoted context omitted.

Collisions can already occur at two connections

AFAIK, all the major tcp/ip stacks do round robin port assignment. Given relatively short, and relatively few, connections you should not have any collisions.

Imagine the client is actually two machines behind a NAT (corporate network or school) which each open 1 connection.

You can even reproduce this easily with two docker containers on the same host and curl using the "--local-port" option.

Re: AWS NLBs and the mixed up TCP connections

#28

Post by watermelon0 »

benmmurphy wrote: post #11
rocqua wrote: post #6

What would be required for the NLB host to do port-rewriting to make things stable? Is the NLB host not present on the return-path? If that is not the issue, is this a performance issue? Could random allocation of ports work instead?

You can run NLB in ip mode instead of instance mode and I don’t think it will have this issue. In ip mode your server sees the IP address of the NLB instead of the client.

There is no simple way to do this when using Kubernetes. NLB provisioned via Kubernetes will use instance mode, and you cannot change that, and aws-alb-ingress-controller doesn't support NLBs.

Weirdly, provisioning NLB via Kubernetes supports `aws-load-balancer-cross-zone-load-balancing-enabled` annotation, even though this is quite broken behaviour, as per the article.

Re: AWS NLBs and the mixed up TCP connections

#29

Post by benmmurphy »

they could fix the problem by assigning each NLB AZ IP an aliased IP on your target machine. however, this would be an absolute mess to configure.

For example if the 3 NLB AZ IPs were:

1.2.3.6 1.2.3.7 1.2.3.8

then if you set up 3 IP aliases on your target instance:

192.168.2.6 192.168.2.7 192.168.2.8

then if the NLB always mapped:

1.2.3.6 -> 192.168.2.6 1.2.3.7 -> 192.168.2.7 1.2.3.8 -> 192.168.2.8

then there would be no clashes.

Post reply on HN