Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3 Outage

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 142 posts

Re: AWS S3 Outage

#81

Post by kureikain »

hughstephens wrote: post #69
kureikain wrote: post #67

It's two hours. Luckily it happenned around 12:00AM otherwise we are having a bad day. Whomever being a DevOps or SysAdmin probably cannot sleep tonight :(. In our case, we put Fastly on top of our assets/images so that only a partil of request get errors. The cached object on Fastly is still fine.

haha if two hours is nothing to you, that's pretty lucky. We work 24x7 (global customers, yes US is a bit busier but not by a lot) and we're an Australian company, so not a whole lot of fun.

Yeah, sorry to hear that :(. I mean it's already two hours, a long time.

However, if it doesn't come back before 5AM I think we are screw :(

Re: AWS S3 Outage

#82

Post by kureikain »

jbbarth wrote: post #68
kureikain wrote: post #67

It's two hours. Luckily it happenned around 12:00AM otherwise we are having a bad day. Whomever being a DevOps or SysAdmin probably cannot sleep tonight :(. In our case, we put Fastly on top of our assets/images so that only a partil of request get errors. The cached object on Fastly is still fine.

Actually on some parts of the world it's not 12:00AM you know, and there are sysadmins/devops there too :-)

yeah, forgot that :(. Hope it comes back soon.

Re: AWS S3 Outage

#83

Post by ork »

bbrazil wrote: post #56
nmjohn wrote: post #3

I'm seeing it as well - majority of connections are being dropped for us atm The Amazon S3 team recently completed some maintenance changes to Amazon S3’s DNS configuration for the US STANDARD region on July 30th, 2015. You are receiving this email because we noticed that your bucket is still receiving requests on the IP addresses which were removed from DNS rotation. These IP addresses will be disabled on August 10t…

Java by default can cache DNS forever, which may be why many people are seeing problems. Set networkaddress.cache.ttl to adjust this. http://javaeesupportpatterns.blogspot.ie/2011/03/java-dns-ca... has more detail.

Reading this article, it seems that starting at 1.6 the default TTL became 30s.

Re: AWS S3 Outage

#87

Post by nzadrozny »

fizx wrote: post #16

I wonder what it would take for amazon to show one of the yellow icons on their status page? Has it ever happened? Would a datacenter have to fall in the ocean?

Seems that "S3 offline" is the AWS equivalent of a datacenter falling into the ocean. Gotta wonder how many services are using S3 as a faux message queue?

Re: AWS S3 Outage

#88

Post by rplnt »

jakozaur wrote: post #37

Open-source library request: A library that lets you use S3 and Google Storage Cloud simultaneously and fail-back to another if one have problems. There are many use-case when paying 2x for storage is a reasonable tradeoff for higher availability and also be provider independent.

There used to be this https://deltacloud.apache.org/ (not exactly what you want + it seems to be dead).

Re: AWS S3 Outage

#89

Post by mentat »

Current list of additional services affected: CloudSearch

Elastic Compute Cloud

Elastic Load Balancing

Elastic MapReduce

Relational Database Service

CloudTrail

Config

Lambda

OpsWorks

Post reply on HN