41–50 of 67 posts
If the description is correct, then there would be a much worse problem if you had two clients talking to the same NLB. If they happen to use the same local port number at the same time, then it would all fail. This seems likely to happen a lot more than a single client reusing its local-port because it is a different remote IP. How does an NLB handle the following two connections: 1) from 1.2.3.4:44444 to NLB:443 2)…
This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ! If you need to call out of the the AZ for other data or API sources, either figure o…
This is good advice but not always easy to implement. We have some customers that insist on using IPs instead of DNS (usually because of bad/old software on their side). In our case we have some commercial LBs that can pass the EIP to each other as needed. However we do see quite a few resets so I wonder if something like this is still going on.
We reduced the severity of this by randomizing port mappings on the NAT but that just reduces the probability. They claimed that it was a high priority issue for them back then but seems this is still in the wild - insane!
Earlier quoted context omitted.
Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage. You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite…
That's a really interesting point. The startup I currently work for only uses a single AZ due to financial concerns (and some performance as well), but I assume we'll have to move to more AZs for reliability. Would you advise the same for clusters of RDS and Elasticache? I'm wondering how you would even go about having two separate data sources, how would this be manageable?
Does this happen only with IPv4 and not IPv6?
An ALB is the only one of the load balancers in AWS to support IPv6, but only to terminate the connection, not to send traffic to an IPv6 target.
Earlier quoted context omitted.
That's a really interesting point. The startup I currently work for only uses a single AZ due to financial concerns (and some performance as well), but I assume we'll have to move to more AZs for reliability. Would you advise the same for clusters of RDS and Elasticache? I'm wondering how you would even go about having two separate data sources, how would this be manageable?
I've found that for RDS, a writer instance and a hot standby reader instance with automatic failover work pretty well. When a failover happens, you're usually looking at about 30 seconds of downtime, which is "good enough" for most purposes.
This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ! If you need to call out of the the AZ for other data or API sources, either figure o…
Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage. You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite…
Earlier quoted context omitted.
Since you seem knowledgable and I'm not an AWS customer... Can you confirm the general workings of the Network Load Balancer? It seems like it's basically doing NAT to take client:port server:port to client:port instance:port, including adjusting instance NAT mappings so the outgoing traffic gets NATed back? Sort of like direct server return load balancing, but weird because the instance doesnt see the service IP and…
There are two modes to NLB: 'instance' mode which is what they are describing in the article. I think what happens is the packets hit the AWS network and then somehow AWS decides whether it is an existing flow or chooses a target instance to create a new flow. then it just routes the packet by only modifying the destination address. it can do this because the whole of the AWS network is basically a lie and ip packets…
Does the instance port have to match the service port? In instance mode, you could setup so us-east1 is like 441, us-east2 is 442, us-west1 is 451 etc. And then you wouldn't have tcp 4-tuple collisions on the instance.
Similarly for ip mode, you could put the same host in for ports 440-449 and get 10x 55,000 connections, or however many ports you need to hit your numbers.
(Port numbers chosen for example only, check /etc/services if you don't want to stomp on allocations for services you're probably not using)
Thanks!
Edit: actually i just realized for really high number of long conns that wouldn’t matter. Hm maybe not rewrite the dst ip and make vms handle that. Seems like a much more intrusive chancge though