Ask HN: Is S3 down? • Hacker News Board

Re: Ask HN: Is S3 down?

#671

Post by prdonahue » Tue, Feb 28, 2017, 8:56 PM UTC

problems wrote: post #623

emrekzd wrote: post #312

Earlier quoted context omitted.

In December 2015 I received an e-mail with the following subject line from AWS, around 4 am in the morning: "Amazon EC2 Instance scheduled for retirement" When I checked the logs it was clear the hardware failed 30 mins before they scheduled it for retirement. EC2 and root device data was gone. The e-mail also said "you may have already lost data". So I know that Amazon schedules servers for retirement after they alr…

That's completely ridiculous, get some fucking RAID Amazon. I order drives off newegg directly to my DC and I'm yet to lose data with the cheapest drives available in RAID10.

Yes, solving problems at your scale and AWS' are quite comparable.

Re: Ask HN: Is S3 down?

#672

Post by Raphmedia » Tue, Feb 28, 2017, 8:57 PM UTC

yichi wrote: post #605

benwilber0 wrote: post #287

Notice how Amazon.com itself is unaffected. They're a lot smarter than us.

I do recall reading somewhere that Amazon.com isn't actually hosted or fully leveraging on the AWS platform, mostly due to the political struggle between the AWS and the merchant department.

Sales Department: "We require 100% uptime! Can you do that?"

AWS Department: "Wellll, if we don't change the status to red, it's as if we were up all the time!"

Re: Ask HN: Is S3 down?

#673

Post by problems » Tue, Feb 28, 2017, 8:58 PM UTC

prdonahue wrote: post #671

problems wrote: post #623

Earlier quoted context omitted.

That's completely ridiculous, get some fucking RAID Amazon. I order drives off newegg directly to my DC and I'm yet to lose data with the cheapest drives available in RAID10.

Yes, solving problems at your scale and AWS' are quite comparable.

Not saying my scale is the the same at all - but the fact they can't do something so simple that I can do it as a single individual is embarrassing at best.

Simple solutions to this do scale - Linode and DigitalOcean don't have such issues for example - and while they're not Amazon scale, they are quite large and I'd say they prove the concept.

Re: Ask HN: Is S3 down?

#674

Post by samaysharma » Tue, Feb 28, 2017, 8:58 PM UTC

From https://status.aws.amazon.com/: "Update at 12:52 AM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour."

(I think the AM means PM)

Re: Ask HN: Is S3 down?

#675

Post by kyled » Tue, Feb 28, 2017, 8:59 PM UTC

aabajian wrote: post #638

Leap day bug?

And yet people think I'm crazy for wanting to wrap get time functions so code can be tested...

Re: Ask HN: Is S3 down?

#676

Post by uranian » Tue, Feb 28, 2017, 9:00 PM UTC

Heroku seems to suffer from this too

Re: Ask HN: Is S3 down?

#677

Post by all_usernames » Tue, Feb 28, 2017, 9:00 PM UTC

xbryanx wrote: post #135

gamache wrote: post #127

A piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You're signing up for the oldest hardware and the most frequent outages. For legacy customers, it's hard to move regions, but in general, if you have the chance to choose a region other than us-east-1, do that. I had the chance to transition to us-west-2 about 18 months ago and in that time, there have been at least three us-east-1 ou…

I'm getting the same outage in us-west-2 right now.

Our services in us-west-2 have been up the whole time.

I think the problem is globally accessible APIs are impacted. As others have noted, if you can use region/AZ-specific hostnames to connect, you can get though to S3.

CloudFront is faithfully serving up our existing files even from buckets in US-East.

Re: Ask HN: Is S3 down?

#678

Post by AzzieElbab » Tue, Feb 28, 2017, 9:01 PM UTC

Netflix is up. Enjoy

Re: Ask HN: Is S3 down?

#679

Post by Animats » Tue, Feb 28, 2017, 9:01 PM UTC

AWS is claiming that Simple Storage (US Standard) is starting to come back up as of 12:54 PM PST.

Re: Ask HN: Is S3 down?

#680

Post by cyberferret » Tue, Feb 28, 2017, 9:01 PM UTC

Well, at least our decision to split services has paid off. All of our web app infrastructure is on AWS, which is currently down, but our status page [0] is on Digital Ocean, so at least our customers can go see that we are down!

A pyrrhic victory... ;)

[0] - http://status.hrpartner.io

EDIT UPDATE: Well, I spoke too soon - even our status page is down now, but not sure if that is linked to the AWS issues, or simply the HN "hug of death" from this post! :)

EDIT UPDATE 2: Aaaaand, back up again. I think it just got a little hammered from HN traffic.