AWS S3 Outage • Hacker News Board

Re: AWS S3 Outage

#111

Post by bardworx » Mon, Aug 10, 2015, 10:45 AM UTC

jschorr wrote: post #86

gedrap wrote: post #85

I see this thread as a list of services depending on S3 being healthy.

Also a good list of developers and devops people not getting any sleep tonight :-/

Yup!

3am "Dry Run" Staging on Heroku...fail 7am Deployment to Production....fail

Now: "We have confirmed elevated latencies affecting our SendEmail, SendRawEmail and SendSMTPEmail APIs in the US-EAST-1 Region and are working to address the problem."

Which is perfect since most of our PO orders are being placed between 6am-10am.

Tonight, adult beverages will be needed after everything is resolved.

Re: AWS S3 Outage

#112

Post by ownagefool » Mon, Aug 10, 2015, 10:48 AM UTC

mentat wrote: post #65

jschorr wrote: post #55

Latest Update from http://status.aws.amazon.com/ : 1:52 AM PDT We are actively working on the recovery process, focusing on multiple steps in parallel. While we are in recovery, customers will continue to see elevated error rate and latencies.

The EC2 launch thing is still green even though all launches fail for us.

Could you expand what you mean? Is that because your launch is trying to fetch something from s3?

Re: AWS S3 Outage

#113

Post by Perdition » Mon, Aug 10, 2015, 10:51 AM UTC

cperciva wrote: post #22

gschier wrote: post #19

Earlier quoted context omitted.

Ya, it bothers me that their status messages for major outages are simply "elevated error rates".

I'm sure "elevated error rates" is the first alarm which goes off. And once they've put that description onto the status page, they're probably more worried about getting it fixed than going back and changing the wording.

Amazon isn't a couple of guys in a garage. They have hordes of administrative personnel who could be tasked to update a status page.

Companies 1/1000th the size of Amazon can manage it.

Re: AWS S3 Outage

#114

Post by hughstephens » Mon, Aug 10, 2015, 10:54 AM UTC

gotroot wrote: post #91

hughstephens wrote: post #66

Earlier quoted context omitted.

It only goes red when a nuclear event occurs, obliterating most of humanity and only the machines remain.

That would actually turn it green again, I think.

maybe true – but if a server gets no requests, does it really exist?

or would the inter-machine chatter continue ad infinitum? would they run out of IPs or successfully transition to IPv6?

So many questions.

Re: AWS S3 Outage

#115

Post by Xorlev » Mon, Aug 10, 2015, 10:58 AM UTC

ork wrote: post #83

bbrazil wrote: post #56

Earlier quoted context omitted.

Java by default can cache DNS forever, which may be why many people are seeing problems. Set networkaddress.cache.ttl to adjust this. http://javaeesupportpatterns.blogspot.ie/2011/03/java-dns-ca... has more detail.

Reading this article, it seems that starting at 1.6 the default TTL became 30s.

It's not true as far as I can tell. You need this flag set or it keeps DNS entries forever, at least on the JVM we run (Oracle, 1.8u51).

Re: AWS S3 Outage

#116

Post by nebulon » Mon, Aug 10, 2015, 11:03 AM UTC

jschorr wrote: post #55

Latest Update from http://status.aws.amazon.com/ : 1:52 AM PDT We are actively working on the recovery process, focusing on multiple steps in parallel. While we are in recovery, customers will continue to see elevated error rate and latencies.

Went to green again and it seems to be resolved.

Re: AWS S3 Outage

#117

Post by mryan » Mon, Aug 10, 2015, 11:05 AM UTC

hughstephens wrote: post #114

gotroot wrote: post #91

Earlier quoted context omitted.

That would actually turn it green again, I think.

maybe true – but if a server gets no requests, does it really exist? or would the inter-machine chatter continue ad infinitum? would they run out of IPs or successfully transition to IPv6? So many questions.

Cory Doctorow's When Sysadmins Ruled the Earth touches on this - what happens to the internet activity during a global crisis?

http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...

Re: AWS S3 Outage

#118

Post by mryan » Mon, Aug 10, 2015, 11:06 AM UTC

Should be back to normal now. The latest update is:

3:46 AM PDT Between 12:08 AM and 3:40 AM PDT, Amazon S3 experienced elevated error rates and latencies. We identified the root cause and pursued multiple paths to recovery. The error has been corrected and the service is operating normally.

Re: AWS S3 Outage

#119

Post by cperciva » Mon, Aug 10, 2015, 11:19 AM UTC

Perdition wrote: post #113

cperciva wrote: post #22

Earlier quoted context omitted.

I'm sure "elevated error rates" is the first alarm which goes off. And once they've put that description onto the status page, they're probably more worried about getting it fixed than going back and changing the wording.

Amazon isn't a couple of guys in a garage. They have hordes of administrative personnel who could be tasked to update a status page. Companies 1/1000th the size of Amazon can manage it.

Yes and no. Sure, Amazon has administrative personnel. Sure, some of those administrative personnel would probably be happy to get paid extra to carry a pager and be summoned to work at 3AM to update a status page.

But the last thing you want to do is put inaccurate information onto a status page; so mere administrative personnel isn't enough -- you'd need people who understand enough about the system to be able to write about it without introducing errors.

I'm guessing that the intersection of "administrative personnel", "willing to carry pagers" and "understand the internals of AWS services" is a very small set.

Re: AWS S3 Outage

#120

Post by ramon » Mon, Aug 10, 2015, 11:34 AM UTC

This happens a couple of times, specially when replacing files frequently! I submit things to S3 everyday, if you're uploading a chunck of files you'll get errors every now and then when replacing files.