Yep, had issues yesterday. Botched a big deploy for me too.
Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
31–35 of 35 posts
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#32Earlier quoted context omitted.
It always has been - us-east-1 is the biggest region by far, so scale problems tend to arise there first.
All updates and patching start with us-east-1. The main reason it has the most issues is because it's the guinea pig for production update deployments.
That's a repeated wrong assumption. In general there is no unified rollout plan within AWS. Each service can use a different method.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#33Earlier quoted context omitted.
It always has been - us-east-1 is the biggest region by far, so scale problems tend to arise there first.
All updates and patching start with us-east-1. The main reason it has the most issues is because it's the guinea pig for production update deployments.
I think it has more to do with the fact that it’s the oldest region, and they support some odd things there that do not exist in other regions.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#34Earlier quoted context omitted.
That's no longer the case, except in legacy regions (which IAD is); I think there's a reconciliation mechanism though in those cases.
Can you cite something about this? I can't find anything, except confirming that's not true [1]. Your profile says you work at AWS so I assume you have inside info on this. Perhaps you could also explain why this change would get made? I always considered it pretty smart to do - With consistent machines, wouldn't the lowered-lettered zones get significantly more traffic? Most of my deploys go to a+b or a+b+c (and I h…
Every account has the AZ names randomised.
My eu-central-1a is not your eu-central-1a but may be your eu-central-1c.
Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?
#35Earlier quoted context omitted.
That's no longer the case, except in legacy regions (which IAD is); I think there's a reconciliation mechanism though in those cases.
Can you cite something about this? I can't find anything, except confirming that's not true [1]. Your profile says you work at AWS so I assume you have inside info on this. Perhaps you could also explain why this change would get made? I always considered it pretty smart to do - With consistent machines, wouldn't the lowered-lettered zones get significantly more traffic? Most of my deploys go to a+b or a+b+c (and I h…
Zone IDs are meant to reconcile this. And due to the nature of my job, it creates plenty of headaches explaining why two accounts can have different AZ names for the 'same' place.
https://docs.aws.amazon.com/ram/latest/userguide/working-wit...
My understanding of the 'why' AWS did this for many regions was to avoid folks hammering the "first" zone they came across (a) when they either didn't care about multi-zone availability or were ignorant of the difference it made. If everyone hops in the 'first' zone, you could end up with disproportionate amounts of traffic. Either way, given how many new regions don't do this (either because they stopped, or that new regions tend to come up slowly one zone at a time), it seems they've abandoned the practice. Unfortunately, even new accounts in these legacy regions still end up with randomized mappings.
I also suspect they didn't care much about this until cross-account features were offered.