Not as much as TCP issues, but increased API call failure, across multiple services (cloudformation, ec2, rds), yes, on us-east-1. Mind you, still pretty low rate, but enough to notice some pattern.
Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
21–30 of 35 posts
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#22We are seeing sporadic connection issues where tcp syn packets are dropped before reaching our elb. Have noticed off and on for a few weeks now. Still investigating and have support ticket out with aws.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#23Earlier quoted context omitted.
That's no longer the case, except in legacy regions (which IAD is); I think there's a reconciliation mechanism though in those cases.
Can you cite something about this? I can't find anything, except confirming that's not true [1]. Your profile says you work at AWS so I assume you have inside info on this. Perhaps you could also explain why this change would get made? I always considered it pretty smart to do - With consistent machines, wouldn't the lowered-lettered zones get significantly more traffic? Most of my deploys go to a+b or a+b+c (and I h…
> Perhaps you could also explain why this change would get made?
I'm not related to Amazon/AWS (I'm not even a customer of AWS), but if it were my decision, I would want the mappings to be the same for everyone, because customer specific mappings makes it harder for customers to coordinate.
Ex: if I'm your customer and we're both on AWS and want to have the best experience between servers, I'd like to put my VMs that talk to yours in the availability zones you're in.
If I want to have the highest availability talking to your servers, again, I want to be in the same zones as you --- or at least if I'm in -1a and you're in -1b, I should expect to need to go to a different geo from time to time if only one AZ has an incident.
This is a lot harder if some of the identifiers mean different things.
Re: balancing. Thehre's usually ways to influence people. Default choices in the console, asking big customers to put new VMs in other zones or rebalance, spot prices, etc. I've been a big enough customer to see that at other hosting. If the top 10 customers control 50% of the VMs (made up statistic, but I was a #1 customer controlling 50% of a DCs traffic, so plausible), you only need to have a few conversations to get things moving.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#24Perhaps related: our load tests this week showed an increase in 502s from the ALB. The app server request logs indicate those requests never made it from the ALB.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#25Earlier quoted context omitted.
All updates and patching start with us-east-1. The main reason it has the most issues is because it's the guinea pig for production update deployments.
Hmm that’s not true, nowadays most new things will start with small regions, typically us-east-2
+1
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#26Earlier quoted context omitted.
It always has been - us-east-1 is the biggest region by far, so scale problems tend to arise there first.
All updates and patching start with us-east-1. The main reason it has the most issues is because it's the guinea pig for production update deployments.
Not true at all
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#27One instance was randomly powered down about 22 hours ago.
We saw a synthetic monitor failure at midnight. Investigation of the transaction trace shows that a specific code path that should take maybe ~100ms took almost 40000ms.
It could have been unresponsive EBS. Or failure to look up the Redis server's IP address. Or some other infrastructure-level failure. The synthetic browser saw it as a 502.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#28We are seeing sporadic connection issues where tcp syn packets are dropped before reaching our elb. Have noticed off and on for a few weeks now. Still investigating and have support ticket out with aws.
Do ELBs operate behind the 'security groups' firewall? I don't think they do, but if they do, you might be hitting connection tracking limits? It's consistent with dropped syns.
Some more details below, but if you use normalish (naive?) rules in the aws firewall, you get connection tracking behavior and there's an unspecified connection limit for each instance type. Above that limit, incoming syns are dropped.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-secu...
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#29Earlier quoted context omitted.
It always has been - us-east-1 is the biggest region by far, so scale problems tend to arise there first.
All updates and patching start with us-east-1. The main reason it has the most issues is because it's the guinea pig for production update deployments.
Do you really believe that the biggest region is used for testing?
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#30We had a few minutes earlier this week where a machine saw packets in/out go to zero for no discernible reason.
Same here in us-east-2.