Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#1

Post by bognition » Fri, Aug 14, 2020, 4:42 PM UTC

Over the last week we've seen random TCP connection issues in the US-1 East data center in AWS. Has anyone else been seeing this?

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#2

Post by osipovas » Fri, Aug 14, 2020, 5:00 PM UTC

Maybe AWS US East is going to be the region with random failures?

"I want to have an AWS region where everything breaks with high frequency" https://news.ycombinator.com/item?id=24103746

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#3

Post by [deleted] » Fri, Aug 14, 2020, 5:01 PM UTC

[deleted]

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#4

Post by WaxProlix » Fri, Aug 14, 2020, 5:04 PM UTC

In any specific AZ or across the board?

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#5

Post by BookPage » Fri, Aug 14, 2020, 5:05 PM UTC

It would help if you defined exactly what type of issues you experienced. Packet loss? Early RSTs? Latency? Single AZ or cross AZ? Same for VPCs or NAT'd internet traffic?

You should take some tcpdumps and open a support case.

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#6

Post by choxi » Fri, Aug 14, 2020, 5:06 PM UTC

osipovas wrote: post #2

Maybe AWS US East is going to be the region with random failures? "I want to have an AWS region where everything breaks with high frequency" https://news.ycombinator.com/item?id=24103746

From the comments:

> Isn’t us-east-1 exactly that?

We've come full circle

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#7

Post by bognition » Fri, Aug 14, 2020, 5:14 PM UTC

BookPage wrote: post #5

It would help if you defined exactly what type of issues you experienced. Packet loss? Early RSTs? Latency? Single AZ or cross AZ? Same for VPCs or NAT'd internet traffic? You should take some tcpdumps and open a support case.

Connection issues between hosts, it appears to be across AZ. Internal traffic inside a VPC

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#8

Post by BillinghamJ » Fri, Aug 14, 2020, 5:18 PM UTC

osipovas wrote: post #2

Maybe AWS US East is going to be the region with random failures? "I want to have an AWS region where everything breaks with high frequency" https://news.ycombinator.com/item?id=24103746

It always has been - us-east-1 is the biggest region by far, so scale problems tend to arise there first.

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#9

Post by sudhirj » Fri, Aug 14, 2020, 5:25 PM UTC

WaxProlix wrote: post #4

In any specific AZ or across the board?

Each account's AZ codes point to a different AZ—it's a way of avoiding conversations like this where people think a particular AZ is better than others.

Re: Ask HN: Has anyone else seen TCP connection issues in AWS US East this week?

#10

Post by WaxProlix » Fri, Aug 14, 2020, 5:27 PM UTC

sudhirj wrote: post #9

WaxProlix wrote: post #4

In any specific AZ or across the board?

Each account's AZ codes point to a different AZ—it's a way of avoiding conversations like this where people think a particular AZ is better than others.

That's no longer the case, except in legacy regions (which IAD is); I think there's a reconciliation mechanism though in those cases.