Live data from Hacker News

Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

cloudrail.com

61–69 of 69 posts

Re: Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

#61

Isn't this what Hashicorp's Terraform also provides?

Not quite.

Terraform abstracts the APIs of multiple cloud providers into HCL, HashiCorp Configuration Langauge. What Terraform is essentially doing is enabling you to skip the part where you write code to talk to AWS' API, allowing you to spin an EC2 instance, or talk to GCE and store something in GCS, and instead just work in one language, via one tool. If the underlaying APIs to those providers shift and change over time, HashiCorp updates the logic under the hood within Terraform, and your code continues working. What it does not do is provide a single resource type or function/method that you can use to upload objects to both AWS and GCE - you have to write two separate resources to work with each of the two example cloud providers.

What CloudRail is saying is: the upload() function will work whether you tell it to push the object to S3 or GCS.

What Terraform forces you to do is change `resource "aws_s3_bucket_object" "picture" {}` to `resource "google_storage_bucket_object" "picture" {}` when you want to change from uploading to GCE from AWS.

Make sense? :)

Re: Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

#62

Earlier quoted context omitted.

It's so bad that big companies, like Netflix, have to write a whole service to store a copy of the s3 metadata: http://techblog.netflix.com/2014/01/s3mper-consistency-in-cl... Netflix have turned the S3 metadata eventually consistent problem into s3mpr metadata eventually consistent problem. The difference is that they can now inspect and reason about s3mpr's metadata. Spotify have had to do the same thing for Google…

GCS is a bit better, since individual objects are strongly consistent, but I've been bitten by the eventually consistent listing.

Also on GCS, if you do a HEAD after DELETE on a bucket that is under lifecycle management it returns 200 instead of 404. Not really a consistency issue but it can really come and bite if you if you're not aware of it. GET returns 404 but HEAD returns 200.

I reported it as a bug but Google said it was by design. More specifically they said: "You are correct, if the versioning enabled in your bucket then the object metadata is saved as an archive object in the bucket [1].This is the reason you are getting 200 for your HEAD request."

Re: Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

#63
post #47
post #21

S3Proxy offers similar multi-protocol access at the S3 API layer: https://github.com/andrewgaul/s3proxy It has compatibility with local filesystem, Backblaze B2, Google Cloud Storage, Microsoft Azure, and OpenStack Swift. Note that this is software and not a service. [S3Proxy author plug]

is this compatible with django-storages (which supports S3)

S3Proxy uses Apache jclouds underneath which has broad compatibility with object stores including S3 clones. However no two S3 implementations are alike so you will need to test.

Re: Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

#64
post #21

S3Proxy offers similar multi-protocol access at the S3 API layer: https://github.com/andrewgaul/s3proxy It has compatibility with local filesystem, Backblaze B2, Google Cloud Storage, Microsoft Azure, and OpenStack Swift. Note that this is software and not a service. [S3Proxy author plug]

>Note that this is software and not a service.

It appears that Cloudrail not a service either, just a commercially licensed library (like Qt for example). In fact, I find their simple library approach a bit less awkward than your local server approach, though I suppose that was necessary in order to make a neat drop-in replacement for code that's already targeting S3.

Re: Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

#65
post #16

Do all these services provide equivalent or abstractable guarantees? Amazon S3, for example, provides 'read-after-write' consistency, meaning once you've received a positive response to a put operation, you can expect to be immediately able to retrieve that object. But it used to be 'eventually consistent', meaning it was possible to receive a positive response to a put, but then not be able to read the object immedi…

Very good question. Of course all services behave a little differently which is out of our control. To be honest, I don't have an exact answer on that. In my first tests I could chain a download directly after the upload with all services. But I don't know if that's because they're all guaranteed to be read-after-write or some were eventually-consistent just really fast. In general, if one service is read-after-write…

That's kind of the point though. This abstraction seems doomed to be extremely leaky. Honestly, I'd rather just deal with them separately than be constantly fighting an API.

Re: Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

#69
post #67

Happy to see you guys having success! I've done my master at the University of Mannheim. Never thought that this would happen, but I kind of miss the city.

Thanks. Viele Grüße aus Mannheim!

Viele Grüße aus Passau! :)
Post reply on HN