Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is S3 down?

news.ycombinator.com

Post reply on HN
Print view View original

951–960 of 1001 posts

Re: Ask HN: Is S3 down?

#951

Post by justicezyx »

twakefield wrote: post #833
boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

The brilliance of open sourcing Borg (aka Kubernetes) is evident in times like these. We[0] are seeing more and more SaaS companies abstract away their dependencies on AWS or any particular cloud provider with Kubernetes. Managing stateful services is still difficult but we are starting to see paths forward [1] and the community's velocity is remarkable. K8s seems to be the wolf in sheep's clothing that will break AW…

K8s is a better borg! It leaps forward and build upon many years experience of operating the system.

Re: Ask HN: Is S3 down?

#952

Post by doubleorseven »

dstroot wrote: post #940
eknkc wrote: post #832

Earlier quoted context omitted.

We were on GCP for around a year, it was my decision I really wanted to love GCP and I initially did. But we recently switched to AWS. I think there is little GCP does better than AWS. Pricing is better on paper, but performance per buck seems to be on par. Stability is a lot worse on GCP, and I don't just mean service outages like this one (which they had their fair share) but also individual issues like instances s…

I, for one, would love to see Redis and Postgres on GCP. That would be enough to get me to switch I think.

Wait for the NEXT event. Radis will be announced.

Re: Ask HN: Is S3 down?

#953

Post by anotherturn »

advisedwang wrote: post #849
JPKab wrote: post #753

Earlier quoted context omitted.

I've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google Cloud convert. (Before that, you guys didn't at all have your shit together when it came to customer support) It's not in bad taste, despite other comments saying otherwise. We need to recognize that competition is good, and Amazon isn't the answer to everything.

I work in GCP support. I'm really curious: what do you feel changed that led to such improved support? I'd like to make sure we keep doing it.

Chiming in as I noticed the change too. For a long time it was almost impossible to speak with a human - every query was directed to the extensive but often useless support pages. If a human did respond it often seemed like they weren't savvy enough to handle a microwave let alone solve infra issues.

Then, about a year or two ago - humans actually started responding to and fixing problems. A welcome change!

Re: Ask HN: Is S3 down?

#954

Post by adjkant »

snackai wrote: post #904
JPKab wrote: post #753

Earlier quoted context omitted.

I've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google Cloud convert. (Before that, you guys didn't at all have your shit together when it came to customer support) It's not in bad taste, despite other comments saying otherwise. We need to recognize that competition is good, and Amazon isn't the answer to everything.

What indeed is bad taste is your choice of Google Cloud over AWS. No I really like GCP, use it at core of many apps, but if people really want a decentralized web we need to use more than one provider. Don't "convert". Use booth, redundancy ffs.

What's the point of decentralizing by putting 50% on one, 50% on the other, and no overlap of the groups? You used the word redundancy, but who is willing to actually do that true redundancy?

Re: Ask HN: Is S3 down?

#955

Post by jamesblonde »

jamesblonde wrote: post #919
JPKab wrote: post #753

Earlier quoted context omitted.

I've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google Cloud convert. (Before that, you guys didn't at all have your shit together when it came to customer support) It's not in bad taste, despite other comments saying otherwise. We need to recognize that competition is good, and Amazon isn't the answer to everything.

I just wrote a piece reflecting on the s3 outage and the limitations of s3 metadata/replication: https://medium.com/@jim_dowling/reflections-on-s3s-architect...

Or discuss it here: https://news.ycombinator.com/item?id=13760251

Re: Ask HN: Is S3 down?

#956

Post by ianamartin »

ec109685 wrote: post #888
ianamartin wrote: post #865

This is why it's important to write code that doesn't depend on only a single service provider. S3 is great. But it's better to set up a Riak cluster on AWS than to actually use S3, if you can. The only services my team uses directly are EC2 and RDS, and I'm thinking of moving RDS over to EC2 instances. We are entirely portable. We can move my entire team's infrastructure to a different cloud host really quickly. Our…

Not knowing your situation exactly, but there could be a cost of running your own infrastructure and not taking advantage of their services? For example, are the chances of losing data higher in riak (take into account disaster and operational bugs that could result in data loss or availability issues) than in one of amazon's supported data stores. I don't necessarily disagree with what you are saying but there is co…

Yes. There's a cost to every decision you make. Sometimes it's the cost of s3 being down. Sometimes it's the cost of some developer time to make services agnostic. The value of that isn't immediately obvious, perhaps.

But the developer cost here (my time) was worth it. Our shit wasn't down, while everyone else's was.

I also want to point out that I spent minimal time setting this up. We can deploy to GCE or commodity VPCs at a moment's notice, and that a project I did over a couple of weekends piggybacking on the ansible playbooks I wrote for AWS.

It's not that hard. You have to get your developers on board with being provider agnostic, and you have to be agnostic yourself. But it is not insurmountable.

It also help when you're the lead dev or your team and also have a good relationship with the devops guy. :)

Re: Ask HN: Is S3 down?

#957

Post by metafunctor »

cowkingdeluxe wrote: post #926
metafunctor wrote: post #846

Based on reports from the field, it looks like S3 was down for about three hours for most of their customers. S3 promises four nines of availability (11 nines of durability), so today we got about 3-4 years worth of downtime in one fell swoop. Oops.

Where do you see four nines for S3 SLA? https://aws.amazon.com/s3/sla/ shows 99.9%

It was from memory. I distinctly remember it being advertised as "four nines". Perhaps they've adjusted their marketing. The S3 FAQ still says [1]: "S3 Standard is designed for 99.99% availability".

[1]: https://aws.amazon.com/s3/faqs/

Re: Ask HN: Is S3 down?

#958

Post by qj_li »

013a wrote: post #929
twakefield wrote: post #833

Earlier quoted context omitted.

The brilliance of open sourcing Borg (aka Kubernetes) is evident in times like these. We[0] are seeing more and more SaaS companies abstract away their dependencies on AWS or any particular cloud provider with Kubernetes. Managing stateful services is still difficult but we are starting to see paths forward [1] and the community's velocity is remarkable. K8s seems to be the wolf in sheep's clothing that will break AW…

Is there any way built in to Kubernetes to go multi-AZ, multi-region, or even multi-cloud? Is federation the answer to this? I remember reading somewhere in the K8s documentation that it is designed such that nodes in a single cluster should be as close as possible, like in the same AZ.

Yes, see the blog http://blog.kubernetes.io/2016/07/cross-cluster-services.htm...

Re: Ask HN: Is S3 down?

#959

Post by Taek »

matt4077 wrote: post #900
bbcbasic wrote: post #858

Earlier quoted context omitted.

Oddly, the self promo worked for a Google employee (see the top rated comment by boulos), but not for you.

The google guy pointed out that they are API-compatible (thus easy to run side-by-side) and, most import IMHO, he was humble, self-deprecating, and honest. This here is insane. The claim that something nobody has ever heard of is more reliable than AWS or Google is the worst possible way to promote the product – even if true.

Perhaps I did a bad job of explaining, but it's a fundamentally different architecture, based on the same technology as Bitcoin. It's a platform that's been designed to outlive its parent company. And many blockchains already have outlived the companies and developers that built them. This one just happens to focus on cloud storage.

Re: Ask HN: Is S3 down?

#960

Post by NiekvdMaas »

boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services:

Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue fixed. During the downtime, there is nothing you can do to speed this up - you're waiting helplessly. Because Cloud SQL is a hosted service, you can not connect to a shell or access any filesystem data directly - there is nothing you can do, other than wait for the Google engineers to resolve the problem.

When the Cloud SQL instance was up&running again, support confirmed that there is nothing you can do to prevent a filesystem crash, it "just happens". The workaround they offered is to have a failover set up, so it can take over in case of downtime. The worst part is that GCS refused to offer credit, as according to their SLA this is not considered downtime. The SLA [1] states: "with respect to Google Cloud SQL Second Generation: all connection requests to a Multi-zone Instance fail" - so as long as the SQL instance accepts incoming connections, there is no downtime. Your data can get lost, your database can be unusable, your whole system might be down: according to Google, this is no downtime.

TL;DR: make sure to check the SLA before moving critical stuff to GCS.

[1]: https://cloud.google.com/sql/sla

Post reply on HN