Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is S3 down?

news.ycombinator.com

Post reply on HN
Print view View original

961–970 of 1001 posts

Re: Ask HN: Is S3 down?

#961

Post by hmottestad »

kaishiro wrote: post #935
cperciva wrote: post #787

S3 is currently (22:00 UTC) back up. The timeline, as observed by Tarsnap: First InternalError response from S3: 17:37:29 Last successful request: 17:37:32 S3 switches from 100% InternalError responses to 503 responses: 17:37:56 S3 switches from 503 responses back to InternalError responses: 20:34:36 First successful request: 20:35:50 Most GET requests succeeding: ~21:03 Most PUT requests succeeding: ~21:52

Playing the role of the front-ender who pretends to be full-stack if the money is right, can someone explain the switch from internal error to 503 and back? Is that just them pulling s3 down while they investigate?

503 is typically what we see when our proxy can't connect to the backend server. We usually get 500 with internal server error when we've messed up the backend server.

So it's likely that the first 500s were the backend for s3 failing, then they took the failing backends offline causing the load balancers to throw 503 because they couldn't connect to the backend.

Re: Ask HN: Is S3 down?

#962

Post by cperciva »

kaishiro wrote: post #935
cperciva wrote: post #787

S3 is currently (22:00 UTC) back up. The timeline, as observed by Tarsnap: First InternalError response from S3: 17:37:29 Last successful request: 17:37:32 S3 switches from 100% InternalError responses to 503 responses: 17:37:56 S3 switches from 503 responses back to InternalError responses: 20:34:36 First successful request: 20:35:50 Most GET requests succeeding: ~21:03 Most PUT requests succeeding: ~21:52

Playing the role of the front-ender who pretends to be full-stack if the money is right, can someone explain the switch from internal error to 503 and back? Is that just them pulling s3 down while they investigate?

My guess based on the behaviour I've seen is that internal nodes were failing, and the 503 responses started because front-end nodes didn't have any back-end nodes which were marked as "not failing and ready for more requests". When Amazon fixed nodes, they would have marked the nodes as "not failed", at which point the front ends would have reverted to "we have nodes we can send traffic to" behaviour.

Re: Ask HN: Is S3 down?

#963

Post by TekMol »

NiekvdMaas wrote: post #960
boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services: Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue f…

21 hours sounds pretty long to me. What type of data was it and how long would you have waited until you continued with a backup of the data on a different machine?

Re: Ask HN: Is S3 down?

#965

Post by fidget »

NiekvdMaas wrote: post #960
boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services: Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue f…

The GCS being referred to by the GP is Google Cloud Storage, not Cloud Sequel. You really do need failover set up though. That's true for basically any MySQL installation, managed or not.

Re: Ask HN: Is S3 down?

#966

Post by NiekvdMaas »

TekMol wrote: post #963
NiekvdMaas wrote: post #960

Earlier quoted context omitted.

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services: Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue f…

21 hours sounds pretty long to me. What type of data was it and how long would you have waited until you continued with a backup of the data on a different machine?

We were definitely prepared to recover from a backup, but the support team told us: "the issue with the file system will likely persevere over a backup/restore". So this, in combination with the data loss you have when recovering from a backup, means we basically had no choice other than to wait till the issue was resolved.

Re: Ask HN: Is S3 down?

#968

Post by kiallmacinnes »

snackai wrote: post #904
JPKab wrote: post #753

Earlier quoted context omitted.

I've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google Cloud convert. (Before that, you guys didn't at all have your shit together when it came to customer support) It's not in bad taste, despite other comments saying otherwise. We need to recognize that competition is good, and Amazon isn't the answer to everything.

What indeed is bad taste is your choice of Google Cloud over AWS. No I really like GCP, use it at core of many apps, but if people really want a decentralized web we need to use more than one provider. Don't "convert". Use booth, redundancy ffs.

Pity I can't upvote more than once! :)

This whole idea of being angry at a vendor for deprecating something with 1yr notice is just ridiculous!

People need to realize they are choosing lock-in, and are choosing the risk of deprecation every time they decide to use a cloud service with no drop in competition/open source/etc.

Own your choices people, don't blame others...

Re: Ask HN: Is S3 down?

#969

Post by kiallmacinnes »

Svenskunganka wrote: post #892
ehsankia wrote: post #874

Earlier quoted context omitted.

Honestly, if you're a big service that millions of people use, you should not put all your eggs in a single basket and should probably use a mix, in case one of the clouds goes down like in this case.

One of the biggest reasons you go for a cloud is because you don't want to deal with reliability & scaling issues, and there's a premium price attached to that. I think most companies using S3 in this case believed they put their eggs in different baskets when they put their data in there.

I can't believe anyone would have thought a dependency on a single AWS region, or even single service provider, would count as having eggs in different baskets, at least, I really hope nobody could think that!

I suspect though that most people affected deemed the risks and costs of failure low enough to be acceptable, and for many people it still is - even with this outage. But that's a conscious decision, rather than plain ignorance.

Re: Ask HN: Is S3 down?

#970

Post by lbill »

NiekvdMaas wrote: post #960
boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services: Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue f…

Not using a failover is a bold choice (not stupid, just bold). A failover is like a good insurance policy: you pay for it, you hope that you'll never need it, but when shit happens you are very happy to have it!
Post reply on HN