Live data from Hacker News

AWS S3 Outage

news.ycombinator.com

121–130 of 142 posts

Re: AWS S3 Outage

#121
post #117

Earlier quoted context omitted.

maybe true – but if a server gets no requests, does it really exist? or would the inter-machine chatter continue ad infinitum? would they run out of IPs or successfully transition to IPv6? So many questions.

Cory Doctorow's When Sysadmins Ruled the Earth touches on this - what happens to the internet activity during a global crisis? http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...

this is excellent, hadn't seen it before! thx

Re: AWS S3 Outage

#122
post #16

I wonder what it would take for amazon to show one of the yellow icons on their status page? Has it ever happened? Would a datacenter have to fall in the ocean?

Seems that "S3 offline" is the AWS equivalent of a datacenter falling into the ocean. Gotta wonder how many services are using S3 as a faux message queue?

I think most services use the message queue as a message queue (SNS), but everyone has to store their files somewhere.

Re: AWS S3 Outage

#123
post #91

Earlier quoted context omitted.

That would actually turn it green again, I think.

maybe true – but if a server gets no requests, does it really exist? or would the inter-machine chatter continue ad infinitum? would they run out of IPs or successfully transition to IPv6? So many questions.

Eventually the undefined behavior of this chatter will result in unallocated memory slowly churning in the garbage collection of time. Some day one of the sectors of unallocated memory will be executed resulting in a self replicating program. This program will evolve and multiply, pondering on the vastness of the S3verse, forever in search of Root.

Re: AWS S3 Outage

#124
post #115
post #83

Earlier quoted context omitted.

Reading this article, it seems that starting at 1.6 the default TTL became 30s.

It's not true as far as I can tell. You need this flag set or it keeps DNS entries forever, at least on the JVM we run (Oracle, 1.8u51).

The default behavior in Java 7 & 8 is to cache forever when a security manager is installed, and to cache for an implementation-specific period of time when a security manager is not installed. Sans security manager, I see 30s in Java 7 -- reference the DEFAULT_POSITIVE constant in sun.net. InetAddressCachePolicy

Re: AWS S3 Outage

#126
post #90
post #73

Earlier quoted context omitted.

What is the last time that AWS gets two consecutive major outages within, maybe, 30 days?

Last one was 10 days ago. https://news.ycombinator.com/item?id=9980222

+1, Also it was down on leap second. http://mashable.com/2015/06/30/aws-disruption

Re: AWS S3 Outage

#127
post #21

Could this be a reason why Heroku is misbehaving? https://status.heroku.com/incidents/792K

I can't help but be a little surprised that Heroku's entire build system is disabled by an S3 failure in one region. Now I'm unable to add a notice about the issues to my site's HTML...

Since S3 is the defacto artifact delivery system for most people that run on AWS, it's not much of a surprise. For the most part, very isolated incidents aside, S3 is rock solid. Even EC2 relies on S3 for launching non-EBS instances.

Re: AWS S3 Outage

#128
post #65

Earlier quoted context omitted.

The EC2 launch thing is still green even though all launches fail for us.

Could you expand what you mean? Is that because your launch is trying to fetch something from s3?

There was a period in which I couldn't launch instances. Meaning the instance state did not ever reach "running" according to the console and were not responsive to initial ssh attempts. (knife timed out after 5 minutes and the machines were still unavailable after several more)

Re: AWS S3 Outage

#129

Open-source library request: A library that lets you use S3 and Google Storage Cloud simultaneously and fail-back to another if one have problems. There are many use-case when paying 2x for storage is a reasonable tradeoff for higher availability and also be provider independent.

The GCS command line tool, gsutil, can talk to both S3 and GCS. That might be a nice place to start.

Re: AWS S3 Outage

#130

Earlier quoted context omitted.

Could you expand what you mean? Is that because your launch is trying to fetch something from s3?

There was a period in which I couldn't launch instances. Meaning the instance state did not ever reach "running" according to the console and were not responsive to initial ssh attempts. (knife timed out after 5 minutes and the machines were still unavailable after several more)

Maybe related to AMI retrieval from S3?
Post reply on HN