Live data from Hacker News

AWS NLBs and the mixed up TCP connections

niels-ole.com

11–20 of 67 posts

Re: AWS NLBs and the mixed up TCP connections

#11
post #6

What would be required for the NLB host to do port-rewriting to make things stable? Is the NLB host not present on the return-path? If that is not the issue, is this a performance issue? Could random allocation of ports work instead?

You can run NLB in ip mode instead of instance mode and I don’t think it will have this issue. In ip mode your server sees the IP address of the NLB instead of the client.

Re: AWS NLBs and the mixed up TCP connections

#12
On the issue of NATs and to extend this to GCP, we had an issue where the NAT was just dropping SYN packets. Clients would then eventually timeout, but connection pools eventually became fully drained. We had to look at tcpdump on the clients to see what was going on. It was 'solved' by giving machines that made external calls their own External IP. I don't know if it's still an issue.

Re: AWS NLBs and the mixed up TCP connections

#13
I debugged and identified the exact same problem a few weeks ago. I don't have any solutions, but can confirm what you're seeing. I suspect most clients aren't creating enough tcp connections in the window to cause a collision. (In our case, we discovered the issue from our load box during performance testing)

Re: AWS NLBs and the mixed up TCP connections

#14
post #8
post #7

Earlier quoted context omitted.

Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage. You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite…

That's a really interesting point. The startup I currently work for only uses a single AZ due to financial concerns (and some performance as well), but I assume we'll have to move to more AZs for reliability. Would you advise the same for clusters of RDS and Elasticache? I'm wondering how you would even go about having two separate data sources, how would this be manageable?

Active-passive with database replication and manual failover is the usual solution.

Re: AWS NLBs and the mixed up TCP connections

#16
If you have cross zone load balancing disabled and you only have one instance per AZ, how do you ensure that when an instance is down or during a deploy, when not all AZ might have instances, that you don't have downtimes? Cross zone load balancing in NLB seems a must to me if you are constantly deploying new targets.

Re: AWS NLBs and the mixed up TCP connections

#17
post #6

What would be required for the NLB host to do port-rewriting to make things stable? Is the NLB host not present on the return-path? If that is not the issue, is this a performance issue? Could random allocation of ports work instead?

My guess is the network routing is happening in the software defined network AWS creates below the traditional networking layers.

Sometimes it's tempting to think AWS' services are just like traditional appliances you'd stick in the middle of a network path but in reality they're all virtual and run on top of their software defined network in something they call hyperplane.

That means traditional intuition about how packets flow, like needing to be in the path to preserve the IP, may not apply.

It seems like in this case they need to preserve the static public IP that was used on the NLB for the incoming request across the whole transaction but maybe aren't doing that.

Re: AWS NLBs and the mixed up TCP connections

#19

It annoys me to no end when people don't explain their abbreviations... "AZ" means Availability Zone, which is to say: Data Center. So cross-AZ means going to multiple data centers.

AZ is a well-known abbreviation in AWS. For all intents and purposes, it can mean a data center, but I've heard that AZs can span multiple physical data centers (which sort of makes sense -- there's a physical limit to how many servers you can fit in to one physical data center)

Re: AWS NLBs and the mixed up TCP connections

#20

It annoys me to no end when people don't explain their abbreviations... "AZ" means Availability Zone, which is to say: Data Center. So cross-AZ means going to multiple data centers.

AZ is a well-known abbreviation in AWS. For all intents and purposes, it can mean a data center, but I've heard that AZs can span multiple physical data centers (which sort of makes sense -- there's a physical limit to how many servers you can fit in to one physical data center)

Some AZs have five data centers, though I’m not sure that up-to-date numbers are published. A DC and a single AZ are definitely not equatable in this regard.
Post reply on HN