Ask HN: Is S3 down? • Hacker News Board

Re: Ask HN: Is S3 down?

#981

Post by Taek » Wed, Mar 01, 2017, 12:43 PM UTC

dang wrote: post #975

Taek wrote: post #217

Mass outage like this is exactly one of the things we are looking to avoid by building a decentralized storage grid with Sia. Sia are immune to situations like this because data is stored redundantly across dozens of servers around the world that are all running on different, unique configurations. Furthermore, there's no single central point of control on the Sia network. Sia is still under heavy development, but it…

An occasional relevant reference to one's own work is fine on HN, but you've crossed into what this community considers spam, both by doing it too often and by doing it in off-topic places. That will eventually get your account and site banned, so please stop overdoing it, especially if your work is actually the kind of thing the community would find interesting in appropriate doses and places.

Understood. I will tone it back.

Re: Ask HN: Is S3 down?

#982

Post by mabbo » Wed, Mar 01, 2017, 12:47 PM UTC

omni wrote: post #897

mabbo wrote: post #878

Earlier quoted context omitted.

> potential business lost My startup's op team had a great discussion today because of this that basically boils down to "if we hit our sales goals, an incident like this a year from now would end our company". Looks like our plans to start prepping for multi-cloud support will be a higher priority.

> an incident like this a year from now would end our company I'm genuinely curious, what kind of business are you in that a four hour outage would end the company? High frequency trading or something?

You're in the right ball park- services for traders and brokers in the finance industry. A two hour outage during trading hours would be an extinction level event.

Re: Ask HN: Is S3 down?

#983

Post by mabbo » Wed, Mar 01, 2017, 12:48 PM UTC

owenmarshall wrote: post #932

mabbo wrote: post #878

Earlier quoted context omitted.

> potential business lost My startup's op team had a great discussion today because of this that basically boils down to "if we hit our sales goals, an incident like this a year from now would end our company". Looks like our plans to start prepping for multi-cloud support will be a higher priority.

I'd be willing to bet that the effort you put into a multi-cloud solution will be more expensive than you think, and far more brittle in the event of an emergency. It always is.

You're not wrong, but if your customers can't afford an outage, you can't afford not having a fallback plan. Probably worth a few trial runs during low volume hours to be sure.

Re: Ask HN: Is S3 down?

#984

Post by eddieroger » Wed, Mar 01, 2017, 1:33 PM UTC

bananabill wrote: post #781

AtheistOfFail wrote: post #694

Earlier quoted context omitted.

Could be worse, your entire infrastructure could be hosted on Heroku. You don't use S3 but because they do, your entire infrastructure crumbles.

I didn't realize Heroku used s3 until today, when my heroku app failed. Makes me wonder why I'm using heroku instead of just aws.

You're paying Heroku to not have to think about deployment or scale, which is also why their marketplace is successful - who wants to think about managing a database, when two clicks and a few environment variables later you can have one. Heroku is great for devs who don't want to think about ops and can afford to throw money at the problem (it gets really expensive really fast).

Re: Ask HN: Is S3 down?

#985

Post by kfkhalili » Wed, Mar 01, 2017, 2:51 PM UTC

So why did the outage occur?

Re: Ask HN: Is S3 down?

#986

Post by avereveard » Wed, Mar 01, 2017, 2:53 PM UTC

adwf wrote: post #980

NiekvdMaas wrote: post #960

Earlier quoted context omitted.

Apologies if this is too much off-topic, but I want to share an anecdote of some some serious problems we had with GCS and why I'd be careful to trust them with critical services: Our production Cloud SQL started throwing errors that we could not write anything to the database. We have Gold support, so quickly created a ticket. While there was a quick reply, it took a total of 21+ hours of downtime to get the issue f…

That isn't just a Google issue though. You'd have had the exact same trouble with AWS/RDS if you're running with no replica. The lack of filesystem access is a security "feature" for both. If you have no HA setup then you have no recourse but to restore to a new server from backup, or wait for your cloud provider to fix it.

RDS has snapshot backups you can create an instance from iirc so you can self fix this kind of issues.

Sure you get downtime all the same but not the waiting for support to solve an instance crash part

Re: Ask HN: Is S3 down?

#987

Post by stef25 » Wed, Mar 01, 2017, 3:17 PM UTC

andmarios wrote: post #850

boulos wrote: post #735

Disclosure: I work on Google Cloud. Apologies if you find this to be in poor taste, but GCS directly supports the S3 XML API (including v4): https://cloud.google.com/storage/docs/interoperability and has easy to use multi-regional support at a fraction of the cost of what it would take on AWS. I directly point my NAS box at home to GCS instead of S3 (sadly having to modify the little PHP client code to point it to st…

As far as I understand the S3 API of Cloud Storage is meant as a temporary solution until a proper migration to Google's APIs. The S3 keys it produces are tied to your developer account. This means that if someone gets the keys from your NAS, he will have access to all the Cloud Storage buckets you have access to (e.g your employer's). I use Google Cloud but not Amazon. Once I wanted a S3 bucket to try with NextCloud…

> OwnCloud

Kicked the tires, not impressed at all. Notes went missing from the interface could only get them back after manually digging through folders via FTP.

Re: Ask HN: Is S3 down?

#988

Post by unclebucknasty » Wed, Mar 01, 2017, 3:45 PM UTC

avereveard wrote: post #986

adwf wrote: post #980

Earlier quoted context omitted.

That isn't just a Google issue though. You'd have had the exact same trouble with AWS/RDS if you're running with no replica. The lack of filesystem access is a security "feature" for both. If you have no HA setup then you have no recourse but to restore to a new server from backup, or wait for your cloud provider to fix it.

RDS has snapshot backups you can create an instance from iirc so you can self fix this kind of issues. Sure you get downtime all the same but not the waiting for support to solve an instance crash part

Yes, and RDS offers point in time recovery at that.

We've had to use it and can confirm that it works as advertised.

Re: Ask HN: Is S3 down?

#989

Post by unclebucknasty » Wed, Mar 01, 2017, 3:57 PM UTC

kiallmacinnes wrote: post #968

snackai wrote: post #904

Earlier quoted context omitted.

What indeed is bad taste is your choice of Google Cloud over AWS. No I really like GCP, use it at core of many apps, but if people really want a decentralized web we need to use more than one provider. Don't "convert". Use booth, redundancy ffs.

Pity I can't upvote more than once! :) This whole idea of being angry at a vendor for deprecating something with 1yr notice is just ridiculous! People need to realize they are choosing lock-in, and are choosing the risk of deprecation every time they decide to use a cloud service with no drop in competition/open source/etc. Own your choices people, don't blame others...

Sounds great on paper, but this is infrastructure level stuff with real world constraints.

The expectation of stability beyond a year is certainly not unreasonable when you're asking people to build their businesses/infrastructure on your platform.

And, building redundancy across providers can be impractical, owed to learning curve, cost duplication, higher outbound bandwidth costs, effort duplication, solution complexity, etc.

Re: Ask HN: Is S3 down?

#990

Post by zedpm » Wed, Mar 01, 2017, 3:58 PM UTC

zedpm wrote: post #476

Can anyone comment on mitigating issues like this with S3 Cross-region replication? I'm reading up on it now while one of my services is dead in the water.

I did some digging and experimentation, and so far it looks like you could keep a backup bucket in another region and use bidirectional replication [0] to keep the two buckets in sync. If something like this happened again, you could point your app(s) at the bucket in another region and keep accepting data. The objects would eventually get replicated back to the original bucket, and you could cut over again when service was restored. There does seem to be an appreciable replication lag, so you could run into problems during your cut where some objects had not yet been replicated, but your app ought to handle things like that gracefully anyway.

[0] https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html