AWS NLBs and the mixed up TCP connections
niels-ole.com
AWS NLBs and the mixed up TCP connections
1–10 of 67 posts
Re: AWS NLBs and the mixed up TCP connections
#2This is surprising behavior, to say the least.
Re: AWS NLBs and the mixed up TCP connections
#3Wow, what a PITA. Thanks for sharing your hard-won insights in this excelent write-up.
Re: AWS NLBs and the mixed up TCP connections
#4Does this happen only with IPv4 and not IPv6?
Re: AWS NLBs and the mixed up TCP connections
#5This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ!
If you need to call out of the the AZ for other data or API sources, either figure out which AZs that service is using and configure to stay in them, or, make sure to go all the way back out to a well-balanced (for resilience) endpoint.
Re: AWS NLBs and the mixed up TCP connections
#6What would be required for the NLB host to do port-rewriting to make things stable? Is the NLB host not present on the return-path?
If that is not the issue, is this a performance issue? Could random allocation of ports work instead?
Re: AWS NLBs and the mixed up TCP connections
#7This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ! If you need to call out of the the AZ for other data or API sources, either figure o…
Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage.
You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite common.
Re: AWS NLBs and the mixed up TCP connections
#8This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ! If you need to call out of the the AZ for other data or API sources, either figure o…
Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage. You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite…
That's a really interesting point. The startup I currently work for only uses a single AZ due to financial concerns (and some performance as well), but I assume we'll have to move to more AZs for reliability.
Would you advise the same for clusters of RDS and Elasticache?
I'm wondering how you would even go about having two separate data sources, how would this be manageable?
Re: AWS NLBs and the mixed up TCP connections
#9I was expecting this to be about the NLB's strange "lag" in updating its flows, wreaking havoc when it comes to changes in the target group, and possibly also relating to weirdly long delays before starting health checks of newly registered targets. I'm bewildered why this hasn't been more of a problem for others, and also why AWS seem to have kind of silently acknowledged the issue (by not closing them), while not coming up with a fix, even after a year. Am I the only one seeing this problem? Ref: https://github.com/aws/containers-roadmap/issues/470 and https://github.com/aws/containers-roadmap/issues/469
Re: AWS NLBs and the mixed up TCP connections
#10Wow very interesting insights. Thanks for the write up.