AWS's us-east-1 region is experiencing issues

#1

Post by zaltekk » Wed, Mar 09, 2022, 8:59 PM UTC

AWS's us-east-1 region is experiencing issues
health.aws.amazon.com

Re: AWS's us-east-1 region is experiencing issues

#2

Post by mtrunkat » Wed, Mar 09, 2022, 9:02 PM UTC

In our case (Apify.com) there was a complete outage of SQS (15mins+), most likely DNS problems + EC2 instances got restarted probably as a result of an SQS outage.

EDIT: Also AWS Lambda seems to be down and AWS EC2 APIs having a very high error rate and machines slow startup times.

Re: AWS's us-east-1 region is experiencing issues

#3

Post by lyjackal » Wed, Mar 09, 2022, 9:06 PM UTC

noticed issues with SQS for a couple minutes. Errors from java sdk, `com.amazonaws.SdkClientException: Unable to execute HTTP request: sqs.us-east-1.amazonaws.com`

Re: AWS's us-east-1 region is experiencing issues

#4

Post by BigGreenTurtle » Wed, Mar 09, 2022, 9:06 PM UTC

mtrunkat wrote: post #2

In our case (Apify.com) there was a complete outage of SQS (15mins+), most likely DNS problems + EC2 instances got restarted probably as a result of an SQS outage. EDIT: Also AWS Lambda seems to be down and AWS EC2 APIs having a very high error rate and machines slow startup times.

Yep, I saw empty responses for sqs.us-east-1.amazonaws.com for a while. Seems okay now though.

Re: AWS's us-east-1 region is experiencing issues

#5

Post by PeterBarrett » Wed, Mar 09, 2022, 9:06 PM UTC

SQS went down for us in us-east-1 and we lost health checks on instances there. Fully recovered now.

Re: AWS's us-east-1 region is experiencing issues

#6

Post by asah » Wed, Mar 09, 2022, 9:06 PM UTC

us-east-1 again!

Re: AWS's us-east-1 region is experiencing issues

#7

Post by [deleted] » Wed, Mar 09, 2022, 9:08 PM UTC

asah wrote: post #6

us-east-1 again!

[deleted]

Re: AWS's us-east-1 region is experiencing issues

#8

Post by xilni » Wed, Mar 09, 2022, 9:09 PM UTC

This is why you are strongly urged not to rely on one region or AZ.

Re: AWS's us-east-1 region is experiencing issues

#9

Post by csdvrx » Wed, Mar 09, 2022, 9:15 PM UTC

As usual?

Re: AWS's us-east-1 region is experiencing issues

#10

Post by pid-1 » Wed, Mar 09, 2022, 9:17 PM UTC

xilni wrote: post #8

This is why you are strongly urged not to rely on one region or AZ.

Given the total amount of money I've lost due a single AZ being down, it was totally worth it to NOT go multi az or multi region so far.

Multi AZ isn't that hard, but generally requires extra costs (one nat gw per az, etc...)

But multi region in AWS is a royal pain in the ass. Many services (like SSO) do not play well with multi region setups, making things really complicated even if you IaCed your whole stack.