Is anyone experiencing this type of issue when publishing PrivateLink Endpoint Services? Presumably it means you need to deploy services to every AZ in the region you are operating? Is that adequate?
Source IP is not preserved with PrivateLink
AWS NLBs and the mixed up TCP connections
51–60 of 67 posts
Re: AWS NLBs and the mixed up TCP connections
#52why do 34.242.162.234 and 18.203.64.15 both translate to 10.54.1.1?
Are they both registered with the NLB and assigned to the same EC2 instance?
Re: AWS NLBs and the mixed up TCP connections
#53Earlier quoted context omitted.
Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage. You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite…
That's a really interesting point. The startup I currently work for only uses a single AZ due to financial concerns (and some performance as well), but I assume we'll have to move to more AZs for reliability. Would you advise the same for clusters of RDS and Elasticache? I'm wondering how you would even go about having two separate data sources, how would this be manageable?
Before assuming that your reliability would be increased by adding more AZs, verify where the problems of reliability comes from in the first place. I find more times than not, the down times comes from people applying changes, not when you just leave things running like they are. It's only if the AZ or underlying machines has troubles, that you should start thinking of expanding to other AZs
Re: AWS NLBs and the mixed up TCP connections
#54Anyone have any context why the OPs post was removed from Reddit? https://www.reddit.com/r/aws/comments/jfx8af/aws_nlbs_and_th...
I'm fairly certain r/aws is run by AWS/Amazon employees so it's only natural that articles that could be seen as negative are removed from it.
Re: AWS NLBs and the mixed up TCP connections
#55I debugged and identified the exact same problem a few weeks ago. I don't have any solutions, but can confirm what you're seeing. I suspect most clients aren't creating enough tcp connections in the window to cause a collision. (In our case, we discovered the issue from our load box during performance testing)
I also saw this recently, although on another provider. The solution in the article's case (as stated) is to not terminate multiple public IPs across one or more NLB which target the same instance. If you must target the same instance more than once, give that instance an IP address for each public IP.
Re: AWS NLBs and the mixed up TCP connections
#56Is anyone experiencing this type of issue when publishing PrivateLink Endpoint Services? Presumably it means you need to deploy services to every AZ in the region you are operating? Is that adequate?
Source IP is not preserved with PrivateLink
OK that makes sense, as long as the service can't see the original source IP/port there's no collision risk.
Re: AWS NLBs and the mixed up TCP connections
#57I hit this issue a year ago, in a slightly different setup. We were trying to expose an internal ECS Service to other services via an NLB. Things worked great, as long as the sending and receiving services didn't end up on the same EC2 instance. Unfortunately this occurred fairly often, since the ECS scheduler wasn't aware of that bizarre constraint (maybe we could have made it aware, but, that seems like a pretty brittle way to fix things).
Our resolution was to just fall back to an ELB. A bummer in that it didn't support dynamic port registration (the service could only be on a single EC2 instance at a time, since it has a static port), but joyful in that we didn't have spurious failures as an artifact of task packing.
Re: AWS NLBs and the mixed up TCP connections
#58On the issue of NATs and to extend this to GCP, we had an issue where the NAT was just dropping SYN packets. Clients would then eventually timeout, but connection pools eventually became fully drained. We had to look at tcpdump on the clients to see what was going on. It was 'solved' by giving machines that made external calls their own External IP. I don't know if it's still an issue.
gcp cloud nat wasn't really used in the first itrations, it was only introduced at the end of 2018 and "private" networks were only a thing in like middle of 2019 and it is still not the default, especially not on gke (besides that cloud nat is cheaper)
Re: AWS NLBs and the mixed up TCP connections
#59Anyone have any context why the OPs post was removed from Reddit? https://www.reddit.com/r/aws/comments/jfx8af/aws_nlbs_and_th...
What makes you think it was removed? It's currently visible but nobody seems to have commented on it.
Re: AWS NLBs and the mixed up TCP connections
#60This is good advice. Ideally, never blend your AZs, each should be an independent stack. Use 3+ not 2, keeps you honest about an availability strategy instead of a standby failover strategy. In front of them, use DNS geo IP or even basic round robin (with service availability check) to get to the NLB. Behind the NLB, stay in that AZ! If you need to call out of the the AZ for other data or API sources, either figure o…
Agree. From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage. You can survive datacenter (AZ) outage IF you have separate stacks per AZ and don't mix traffic. If you have Kafka cluster spread out in 3 AZ don't get surprised if you just LOWERED your availability because any issue in one AZ makes your stack unstable. And issues in single AZ are quite…
> From 6+ years of experience it seems that we got fouled by the multi-az promise of being able to survive datacenter outage.
You have quite a misunderstanding ...
AWS' "multi-az promise" has always been that they will try to take only one AZ down at a time within a region.
It was never "blend your AZ usage so we can't take one down."
If you don't have a wiki page with some HA architecture diagrams for each of your systems, then you probably don't have HA. Hint: at every company that I've worked at, I drew the first diagrams. Something to think about.