Live data from Hacker News

Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

blog.cloudflare.com

171–180 of 220 posts

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#171
Wow, with global replication by default, this looks absolutely perfect for what I'm currently building, even before taking costs into account.

I'm hoping this means what I think it means, that write latencies will be minimal across the globe, since writes will be persisted and ack'd at the closest region and then eventually consistently propagated to other regions?

If so, curious what would happen in a scenario where a region requests an object that has been persisted at another region but not yet propagated? Will it result in a 404 or is the system smart enough to route the request to the region that has the file at the cost higher latency?

From my research so far into S3's cross region replication, the latter behavior doesn't seem possible out of the box since requests have to specify a single region (S3 experts, please do correct me if I'm wrong), so I'm hoping CloudFlare with its deep expertise in managing a global network can differentiate here. Even if it's not offered out of the box, due to the lack of egress costs, it's a lot more feasible to build in this behavior in the application layer with R2 by just racing requests across several regions and taking the one that resolves first (or at all), so very promising regardless.

Also, would love to hear some numbers on what kinds of write latency to expect. From my experience so far, S3 writes for tiny files in a single region take on the order of 50ms ish even for clients in close physical proximity, which is serviceable for my use case, but seems higher than it needs to be (and every little bit I can shave off on latency helps tremendously for what I'm building). Really looking forward to seeing what the CloudFlare solution is capable of here.

Lastly, S3 didn't advertise and guarantee strong read-after-write consistency for same region read/write until late last year. Will R2 offer this out of the gate?

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#172
post #94

Earlier quoted context omitted.

we will :) we're working to reach 5 GB per upload, with multi-part support.

I personally think uploading massive files is not usually desirable. Better would be an easy way to chunk it and upload and have the server put the file back together, which would increase reliability.

S3 protocol supports multipart upload, which is what you're describing. The client chunks it, the server reassembles.

What I'd like to see is PATCH with multipart/byteranges support. =D

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#173
post #167
post #166

Earlier quoted context omitted.

(stupid question) What does that get me? Does Cloudflare have cheap VMs I can rent with competitive prices? (including data egress fees?) For object storage that I play with, some clients are in the cloud, with most sitting at home behind residential internet.

(approximately) Cloudflare provides a proxy service that you'd use to access your B2 data from home or other cloud without paying for egress. They can do this because it costs almost nothing to move data between B2 and Cloudflare, and then from Cloudflare to almost anywhere. Moving data from B2 to most other places on the internet likely costs them more because Backblaze isn't in a position to negotiate adventagous p…

In addition, you need to use Cloudflare web workers if you want any sort of access controls. (I think this is part of why it makes financial sense for Cloudflare to do this)

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#174
Most cloud object storage can be a good off-site backup. But accessing your data fast and cheap is not easy.

That is why SeaweedFS added a gateway to remote object store. https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot... , which asynchronously write local changes to the cloud. If there are enough local capacity, there should be no egress cost.

Hopefully, this can change the common pattern, to really treat the cloud object store as a backup.

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#175

Wow, with global replication by default, this looks absolutely perfect for what I'm currently building, even before taking costs into account. I'm hoping this means what I think it means, that write latencies will be minimal across the globe, since writes will be persisted and ack'd at the closest region and then eventually consistently propagated to other regions? If so, curious what would happen in a scenario where…

> If so, curious what would happen in a scenario where a region requests an object that has been persisted at another region but not yet propagated? Will it result in a 404 or is the system smart enough to route the request to the region that has the file at the cost higher latency?

It's eventually consistent for global replication, but additionally there should be a consistent index of things that are available in other regions? I suppose that's plausible. Seems to defeat a lot of what they're avoiding.

Nothing stopping you from doing a scatter gather on a 404 base on some heuristic in your application code though.

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#176

I'm trying to figure out the long-con of Cloudflare. Too many free/fantastic products.

Get nerds in now with cool cheaper stuff, later when they're enterprise level decision makers Cloudflare is burned into their brain as the go-to

Of course the other angle is that Cloudflare one day ends up like Google, who ended up like Microsoft, who ended up like IBM

As for right now though, cooooool

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#178
post #170
post #167

Earlier quoted context omitted.

(approximately) Cloudflare provides a proxy service that you'd use to access your B2 data from home or other cloud without paying for egress. They can do this because it costs almost nothing to move data between B2 and Cloudflare, and then from Cloudflare to almost anywhere. Moving data from B2 to most other places on the internet likely costs them more because Backblaze isn't in a position to negotiate adventagous p…

Wow! Cool! Very surprised that Cloudflare wouldn't charge an arm and a leg for such a service... considering they're moving the actual bits. I'm poking around at the Cloudflare website, what's the name of the aforementioned service? What term should I google? I'm ignorant of "modern Cloudflare" -- other than reading their fantastic technical blog, I've never used them in a professional capacity and don't know their p…

'bandwidth alliance'

As far as I recall it's not a service or even any setting you have to set up, just a sort of policy.

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#179

Earlier quoted context omitted.

> 'rclone'[2][3] which allows you to move data from cloud to cloud without involving your own bandwidth Maybe I'm reading this wrong, but the data does pass through the machine where rclone is running. rclone does support remote-to-remote transfers[0], but I believe only for remotes of the same type (ie S3 to S3). [0]: https://rclone.org/docs/#server-side-copy

Does this mean "remotes that speak the S3 protocol", or "remotes that are S3"? The former would require S3 supporting induced requests (so it POSTed the data somewhere), the latter would require a "copy" operation on S3. I don't know which one is supported.

Server side copy only works within the same service provider, not across service providers.

Re: Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees

#180
post #60

Earlier quoted context omitted.

Hey, I'm the PM for R2. Yes to range requests. Current object limit is smaller than that, but we don't have a fundamental restriction there. Shoot me an email to gmckeon [at] cloudflare.

Really excited for R2 Storage. I am wondering if R2 can solve a limitation I was running into with BackBlaze. I had tried using BackBlaze 8 months ago as a much cheaper (especially with B2 and CF's Free Data Transfer partnership) replacement for Amazon S3 and was running into a limitation on B2. I had a scenario where my users can upload images from the browser to BackBlaze. I wanted the ability to control the file n…

I have this exact same use case for an app I’m building and would love an answer as well. I built on S3 as a result. When a product says “full S3 api compatibility” this becomes my question.

Going further I don’t want to become someone else’s back door file repo for illegal shit. So presigned upload urls with an enforced file name and configurability over the size limit and expiration of the presignedpost (both in terms of time and number of files) is pretty important to me. S3 does a good job here.

Post reply on HN