Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is S3 down?

news.ycombinator.com

Post reply on HN
Print view View original

781–790 of 1001 posts

Re: Ask HN: Is S3 down?

#781

Post by bananabill »

AtheistOfFail wrote: post #694
cyberferret wrote: post #680

Well, at least our decision to split services has paid off. All of our web app infrastructure is on AWS, which is currently down, but our status page [0] is on Digital Ocean, so at least our customers can go see that we are down! A pyrrhic victory... ;) [0] - http://status.hrpartner.io EDIT UPDATE: Well, I spoke too soon - even our status page is down now, but not sure if that is linked to the AWS issues, or simply t…

Could be worse, your entire infrastructure could be hosted on Heroku. You don't use S3 but because they do, your entire infrastructure crumbles.

I didn't realize Heroku used s3 until today, when my heroku app failed. Makes me wonder why I'm using heroku instead of just aws.

Re: Ask HN: Is S3 down?

#782

Post by gamache »

movedx wrote: post #764
gamache wrote: post #739

Earlier quoted context omitted.

But now you're talking about added effort. Multi-AZ on AWS is easy and fairly automatic, multi-region (and multi-provider) not so much. It's easy to say things like this, but people who can do ops are not cheap and plentiful.

The only difficult aspect of multi-region use is data replication, which I can confirm is a (somewhat) difficult problem. This issue was with S3 which has an option to automatically replicate data from the bucket's region to another one. It's a check box. A simple bit of logic in the application and you can move between regions with ease. Even data replication has options for this, too. And I work in Ops.

Well, you've explained how to do multi-region in S3. Now let's cover EC2, ELB, EBS, VPC, RDS, Lambda, ElastiCache, API Gateway, and all the other bits of AWS that make up my services. And then we can move on to failover application logic.

Re: Ask HN: Is S3 down?

#783

Post by zoloateff »

boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

boulous not in bad taste at all - happy google convert and gcs user works very well for us ymmv

Re: Ask HN: Is S3 down?

#786

Post by gordon_freeman »

alexleclair wrote: post #12

Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved? [1] https://status.aws.amazon.com/

Looks like they have fixed the issue with their health dashboard now.

From https://status.aws.amazon.com/ : Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

Re: Ask HN: Is S3 down?

#787

Post by cperciva »

S3 is currently (22:00 UTC) back up.

The timeline, as observed by Tarsnap:

    First InternalError response from S3: 17:37:29
    Last successful request: 17:37:32
    S3 switches from 100% InternalError responses to 503 responses: 17:37:56
    S3 switches from 503 responses back to InternalError responses: 20:34:36
    First successful request: 20:35:50
    Most GET requests succeeding: ~21:03
    Most PUT requests succeeding: ~21:52

Re: Ask HN: Is S3 down?

#788

Post by foxylion »

mabramo wrote: post #239

Thanks for sharing. I overheard someone on my team say that a production user is having problems with our service. The team checked AWS status, but only took notice of the green checkmarks. Through some dumb luck (and desire to procrastinate a bit), I opened HN and, subsequently, the AWS status page and actually read the US-EAST-1 notification. HN saves the day.

On thing I learned here. When something seems horribly wrong, check HN first, it may be "global" problem.

Re: Ask HN: Is S3 down?

#789

Post by khc »

sachinag wrote: post #756
boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

Opportunistic, sure. But I did not know about the API interoperability. Given the prices, makes sense to store stuff in both places in case one goes down.

I am surprised more people don't know about it. I get questions like https://github.com/kahing/goofys/issues/158 every now and then and to be fair I don't think they market it well: https://cloud.google.com/storage/docs/migrating

Disclosure: I don't work for google but have an upcoming interview there.

Post reply on HN