AWS S3 Outage

#1

Post by gschier » Mon, Aug 10, 2015, 7:37 AM UTC

Seeing huge numbers of 503s from the S3 API in us-east-1. Anyone else having problems? I only found one other on Twitter: https://twitter.com/cperciva/status/630641484677558273

Re: AWS S3 Outage

#2

Post by gschier » Mon, Aug 10, 2015, 7:39 AM UTC

Lots more complaints on Twitter...

Re: AWS S3 Outage

#3

Post by nmjohn » Mon, Aug 10, 2015, 7:42 AM UTC

I'm seeing it as well - majority of connections are being dropped for us atm

  The Amazon S3 team recently completed some maintenance   
  changes to Amazon S3’s DNS configuration for the US STANDARD region on 
  July 30th, 2015.
    
  You are receiving this email because we noticed that your bucket
  is still receiving requests on the IP addresses which were removed 
  from DNS rotation. These IP addresses will be disabled on August 
  10th at  11:00 am PDT, at which time any requests still using
  those addresses will receive an HTTP 503 response status code.
  
  Applications should use the published Amazon S3 DNS names for 
  US STANDARD: either s3.amazonaws.com or s3-external-2.amazonaws.com
  with their associated time to live (TTL) values. Please refer to 
  our documentation at: 
  http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region 
  for more information on Amazon S3 DNS names.

Something to do with that perhaps? AWS sent us that last thursday

Re: AWS S3 Outage

#4

Post by jtwaleson » Mon, Aug 10, 2015, 7:43 AM UTC

loading fonts on aws pages is slow as hell because of this https://aws.amazon.com/ses/

Re: AWS S3 Outage

#5

Post by jsonperl » Mon, Aug 10, 2015, 7:43 AM UTC

Yep, seeing the same.

Re: AWS S3 Outage

#6

Post by adamtulinius » Mon, Aug 10, 2015, 7:45 AM UTC

Can't pull docker images from the hub either, and their statuspage currently shows S3-problems: https://status.docker.com/

Re: AWS S3 Outage

#7

Post by mrsuprawsm » Mon, Aug 10, 2015, 7:46 AM UTC

From http://status.aws.amazon.com/:

12:36 AM PDT We are investigating elevated errors for requests made to Amazon S3 in the US-STANDARD Region.

Re: AWS S3 Outage

#8

Post by [deleted] » Mon, Aug 10, 2015, 7:46 AM UTC

[deleted]

Re: AWS S3 Outage

#9

Post by cperciva » Mon, Aug 10, 2015, 7:47 AM UTC

nmjohn wrote: post #3

I'm seeing it as well - majority of connections are being dropped for us atm The Amazon S3 team recently completed some maintenance changes to Amazon S3’s DNS configuration for the US STANDARD region on July 30th, 2015. You are receiving this email because we noticed that your bucket is still receiving requests on the IP addresses which were removed from DNS rotation. These IP addresses will be disabled on August 10t…

If it is related, someone screwed up badly. I didn't receive such an email; and I'm seeing the same error rate with IPs I resolved 5 minutes ago.

Re: AWS S3 Outage

#10

Post by rwitoff » Mon, Aug 10, 2015, 7:48 AM UTC

same here. our s3 services are reporting similar 503's and network timeouts. a few of our partners are already down as well with their own 500s. another stormy night in the cloud.