Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A New Hope for Object Storage: R2 Enters Open Beta

blog.cloudflare.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: A New Hope for Object Storage: R2 Enters Open Beta

#31

Post by ianopolous »

taylor12345 wrote: post #29
ianopolous wrote: post #27

Earlier quoted context omitted.

Thanks for the awesome product! I have a few questions: 1. Are the rate limits for all users on a given bucket, or by source IP as some other providers do? 2. What is the plan for the rate limits post beta? 3. Do you support the x-amz-content-sha256 header. i.e. If a put contains that header and the content hash doesn't match is it rejected? 4. What does it mean to support pre-signed URLs? Can't a client pre-sign a U…

I'm also an engineer on R2 team. 1 & 2 - We're reserving some ability to rate-limit during beta as we figure out how best to scale our systems under load. I'll defer to vlovich & greg-m for post-beta plans. 3. We do support that header. Mismatching content hashes will be rejected during write. If you ever experience otherwise, please report it. 4. We only support sigv4 auth via header currently: https://docs.aws.amaz…

Thank you, that makes sense.

So presigned URLs that use auth header are fine (We use those extensively in Peergos, as presigned with an auth query parameter isn't cached by browsers as the query string will change with subsequent requests).

Re: A New Hope for Object Storage: R2 Enters Open Beta

#32

Post by ddorian43 »

MallocVoidstar wrote: post #25
ddorian43 wrote: post #15

Earlier quoted context omitted.

This is the exact case when the "except" will be triggered. He is describing a video/image cdn in his comment.

Here's the post where the CEO said R2 can serve video/image just fine: https://news.ycombinator.com/item?id=28682885

Feel free to build a youtube clone on top of it then.

Re: A New Hope for Object Storage: R2 Enters Open Beta

#33

Post by vlovich123 »

ignoramous wrote: post #6
JonoBB wrote: post #5

Earlier quoted context omitted.

If I remember correctly, there was some discussion around R2 acting as a proxy in front of s3. So, if R2 does not yet have the asset, you would automatically pull it in from S3, and from then on R2 would serve it directly. Is that still on the cards?

Not a Cloudflare employee, but: Yep, Cache Reserve is in private beta https://blog.cloudflare.com/introducing-cache-reserve/

Cache reserve is a similar concept but a bit of a different product (although some people might be happy with the no-code approach that enables). Migration is a bit of a different thing.

Re: A New Hope for Object Storage: R2 Enters Open Beta

#34

Post by vlovich123 »

fjfbsufhdvfy wrote: post #26

What stops an attacker from generating millions or billions of requests to your bucket, generating unaffordable bills from the class B charges? When using the "free" cache product, solving this is Cloudflare's problem - they have a lot of incentives to stop people from being able to do that as it would otherwise cost them money. However, when using the R2 product, Cloudflare sure would be incentivized to do whatever…

I think what you’re looking for is Bot fight mode, which is a product we have that lets you block bots and customize how you react to them. There’s a free tier and capabilities get more powerful as you price things out [1]. I would imagine that when we add support for public buckets, we’ll be thinking about how that integrates with Bot fight mode (e.g. enabling the free tier automatically).

Putting cache in front of public buckets is also something you’ll be able to do.

Rate limiting is very similar to spending limits and, at the scale you’re talking about, they’re just expressing the same concept from different perspectives. R2 is currently naturally rate limited anyway because we can’t go faster but that will obviously evolve as we improve the system. Additionally, you can’t currently have public buckets (at least not without exposing it through a Worker you write). The main use for rate limiting and spending limits is more about being OK choosing to have an outage because of a spike in load (e.g. your project going viral unexpectedly on HN before you’ve cost-optimized it for that kind of scale).

From a motivation/incentives perspective, Cloudflare is a SaaS company focusing on security and trust. We’re never incentivized to have unhappy customers, no matter how big of a bill we can deliver for any given month. Doubly so when that comes because you’re under attack vs you’re having success.

So TLDR:

1. Bot fight mode is probably useful as a DOS mitigation tool when hosting a public bucket.

2. When we add virtual pathing, buckets are hosted under your account domain. Unlike with other providers, people can’t scrape public records to find what buckets you’ve exposed (intentionally or otherwise) to the public.

3. The product is currently naturally rate limited.

4. Rate limiting isn’t a viable solution here because an outage is not meaningfully better from us charging you insane amounts of month for that month. If you get an unexpectedly large bill, please reach out to Cloudflare support.

5. Public buckets don’t actually exist yet.

6. Cloudflare is in the business of security and trust and we’re incentivized to do our best to engender that trust.

Does that speak to your concerns?

Re: A New Hope for Object Storage: R2 Enters Open Beta

#35

Post by fjfbsufhdvfy »

vlovich123 wrote: post #34
fjfbsufhdvfy wrote: post #26

What stops an attacker from generating millions or billions of requests to your bucket, generating unaffordable bills from the class B charges? When using the "free" cache product, solving this is Cloudflare's problem - they have a lot of incentives to stop people from being able to do that as it would otherwise cost them money. However, when using the R2 product, Cloudflare sure would be incentivized to do whatever…

I think what you’re looking for is Bot fight mode, which is a product we have that lets you block bots and customize how you react to them. There’s a free tier and capabilities get more powerful as you price things out [1]. I would imagine that when we add support for public buckets, we’ll be thinking about how that integrates with Bot fight mode (e.g. enabling the free tier automatically). Putting cache in front of…

Sounds good! Yeah, my concerns are more related to public buckets, I suppose, and those aren't out.

I have some more thoughts on those:

- Will adding the cache in front of a public bucket come with the necessary ToS updates to allow serving the media content in the bucket?

- It would be nice if there could be some kind of timed authorization for downloads (like a signed token in a header or something?) that does not count towards class B charges if it fails. Currently you can implement this with a worker, but it actually still causes a charge to run the worker, so is mostly ineffective.

Re: A New Hope for Object Storage: R2 Enters Open Beta

#36

Post by vlovich123 »

fjfbsufhdvfy wrote: post #35
vlovich123 wrote: post #34

Earlier quoted context omitted.

I think what you’re looking for is Bot fight mode, which is a product we have that lets you block bots and customize how you react to them. There’s a free tier and capabilities get more powerful as you price things out [1]. I would imagine that when we add support for public buckets, we’ll be thinking about how that integrates with Bot fight mode (e.g. enabling the free tier automatically). Putting cache in front of…

Sounds good! Yeah, my concerns are more related to public buckets, I suppose, and those aren't out. I have some more thoughts on those: - Will adding the cache in front of a public bucket come with the necessary ToS updates to allow serving the media content in the bucket? - It would be nice if there could be some kind of timed authorization for downloads (like a signed token in a header or something?) that does not…

Just to be clear. Putting cache in front of a public bucket has a lot of technical challenges. I don't know when that will become a thing but presumably when it does we'd figure out what the ToS changes would look like.

> that does not count towards class B charges if it fails

What does failure mean in this context?

Re: A New Hope for Object Storage: R2 Enters Open Beta

#37

Post by fjfbsufhdvfy »

vlovich123 wrote: post #36
fjfbsufhdvfy wrote: post #35

Earlier quoted context omitted.

Sounds good! Yeah, my concerns are more related to public buckets, I suppose, and those aren't out. I have some more thoughts on those: - Will adding the cache in front of a public bucket come with the necessary ToS updates to allow serving the media content in the bucket? - It would be nice if there could be some kind of timed authorization for downloads (like a signed token in a header or something?) that does not…

Just to be clear. Putting cache in front of a public bucket has a lot of technical challenges. I don't know when that will become a thing but presumably when it does we'd figure out what the ToS changes would look like. > that does not count towards class B charges if it fails What does failure mean in this context?

Failed to verify the token, as in the user did not have valid authorization to download the file.

Something like this would really help cutting down on unwanted bot requests as each valid request could be traced back to a user and abusive users could be banned.

But if the "failed" requests still cause a charge on your account, this wouldn't really work as intended.

Just trying to think of reliable ways to defend against bot downloading causing massive charges, but also without letting them take down the entire site by triggering something like a billing cap. Hostile world out there...

Re: A New Hope for Object Storage: R2 Enters Open Beta

#38

Post by eins1234 »

vlovich123 wrote: post #23
eins1234 wrote: post #22

Got any examples for https://github.com/aws/aws-sdk-js-v3 or https://github.com/mhart/aws4fetch ? I haven't been able to get either working: const s3Client = new S3Client({ endpoint: 'https://my-account-id.r2.cloudflarestorage.com', credentials: secrets.r2Credentials, }) await s3Client.send( new PutObjectCommand({ Bucket: 'my-bucket, Key: 'test', Body: JSON.stringify({hello: 'world'}), }), ) ^ This fails with a 500 c…

We have several examples [1]. I think you need to set s3ForcePathStyle to true for the first example (or however you do it in the v3 version of the SDK[2]). For the latter, we don't support pre-signed requests. Feel free to ask on discord [3] too. Might get more timely responses. [1] https://developers.cloudflare.com/r2/examples [2] https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clien... [3] https://discord.g…

Thanks for the resources! I managed to get both working in the end. Some notes for future readers:

For SDK v3, I just needed to add the forcePathStyle flag.

For aws4fetch (note that this isn't a presigned request, it's just signing the request for Sig V4 like the official SDK would), I needed to change the fetch url to https://my-account-id.r2.cloudflarestorage.com/my-bucket/key.

Re: A New Hope for Object Storage: R2 Enters Open Beta

#39

Post by vlovich123 »

fjfbsufhdvfy wrote: post #37
vlovich123 wrote: post #36

Earlier quoted context omitted.

Just to be clear. Putting cache in front of a public bucket has a lot of technical challenges. I don't know when that will become a thing but presumably when it does we'd figure out what the ToS changes would look like. > that does not count towards class B charges if it fails What does failure mean in this context?

Failed to verify the token, as in the user did not have valid authorization to download the file. Something like this would really help cutting down on unwanted bot requests as each valid request could be traced back to a user and abusive users could be banned. But if the "failed" requests still cause a charge on your account, this wouldn't really work as intended. Just trying to think of reliable ways to defend agai…

If you gave the user an invalid token (as in R2 can reject it because the token isn't valid), we certainly wouldn't bill (after all, that would be a trivial way someone could launch billing attacks on your account). Now if you gave them a token that validates from our end, then I'm not sure how we'd solve that. I think what you're thinking of is maybe some kind of external IAM integration where you could add an additional authentication layer to whitelabel R2 (but then you're back to the problem where you'd be running a worker to validate the account-level logic and paying for that request).

I think the solution is bot fight protection services from Cloudflare. There's a bunch in there also to allow well-behaved bots to have access and lets you control what those look like.

Post reply on HN