Cloudflare has been attacking the S3 egress problem by creating Sippy: https://developers.cloudflare.com/r2/data-migration/sippy/ It allows you to incrementally migrate off of providers like S3 and onto the egress-free Cloudflare R2. Very clever idea. He calls R2 an undiscovered gem and IMO this is the gem's undiscovered gem. (Understandable since Sippy is very new and still in beta)
From S3 to R2: An economic opportunity
51–60 of 183 posts
Re: From S3 to R2: An economic opportunity
#52Cloudflare may well be on their way to becoming a monopoly, but they certainly show they don't care about abuse. Even if it weren't a simple matter of principle, in case they aren't successful in forcing themselves down everyone's throats, I wouldn't want to host anything on any service that hosts phishers and scammers without even a modicum of concern.
Re: From S3 to R2: An economic opportunity
#53The other hidden cost when you are working with data hosted on S3 is the LIST requests. Some of the data tools seem very chatty with S3, and you end up with thousands of them when you have small filed buried in folders with a not insignifcant cost. I need to dig into it more, but they are always up there towards the top of my AWS bills.
Re: From S3 to R2: An economic opportunity
#54Earlier quoted context omitted.
What are the economics that Amazon and other providers have egress fees and R2 doesn't? Is it acting as a loss leader or does this model still make money for CloudFlare?
also, egress fees are a sort of vendor lock-in, because getting data out of the cloud is vastly more expensive then putting new data into the cloud..
Of course you can (like Snap) but it's a MASSIVE engineering effort and initial expense.
Re: From S3 to R2: An economic opportunity
#55OP is missing that a correct implementation of Databricks or Snowflake will have those instances are running inside the same AWS region as the data. That's not to say R2 isn't an amazing product, but the egregious costs aren't as high since egress is $0 on both sides.
Re: From S3 to R2: An economic opportunity
#56Earlier quoted context omitted.
What are the economics that Amazon and other providers have egress fees and R2 doesn't? Is it acting as a loss leader or does this model still make money for CloudFlare?
Amazon doesn't have unit cost for egress. They charge you for the stuff you put through their pipe, while paying their transit providers only for the size of the pipe (or more often, not paying them anything since they just peer directly with them at an exchange point). Amazon uses $/gb as a price gouging mechanism and also a QoS constraint. Every bit you send through their pipe is basically printing money for them,…
Re: From S3 to R2: An economic opportunity
#57OP is missing that a correct implementation of Databricks or Snowflake will have those instances are running inside the same AWS region as the data. That's not to say R2 isn't an amazing product, but the egregious costs aren't as high since egress is $0 on both sides.
Re: From S3 to R2: An economic opportunity
#58Earlier quoted context omitted.
Amazon doesn't have unit cost for egress. They charge you for the stuff you put through their pipe, while paying their transit providers only for the size of the pipe (or more often, not paying them anything since they just peer directly with them at an exchange point). Amazon uses $/gb as a price gouging mechanism and also a QoS constraint. Every bit you send through their pipe is basically printing money for them,…
You don't get charge for transit if you are sending stuff IN from the internet or to any other AWS resource in that region. So there is no QOS constraint inside except for perhaps paying for the S3 GET/SELECT/LIST costs. It is pretty much exclusively to lock you into their services. It heavily impacts multi-cloud and outside of AWS service decisions when your data lives in AWS and is taxed at 5-9 cents a GB to come o…
Re: From S3 to R2: An economic opportunity
#59> In fact, there’s an opportunity to build entire companies that take advantage of this price differential and I expect we’ll see more and more of that happening. Interesting. What sort of companies can take advantage of this?
I work for a non-profit doing digital preservation for a number of universities in the US. We store huge amounts of data in S3, Glacier and Wasabi, and provide services and workflows to help depositors comply with legal requirements, access controls, provable data integrity, archival best practices, etc.
There are some for-profits in this space as well. It's not a huge or highly profitable space, but I do think there are other business opportunities out there where organizations want to store geographically distributed copies of their data (for safety) and run that data through processing pipelines.
The trick, of course, is to identify which organizations have a similar set of needs and then build that. In our case, we've spent a lot of time working around data access costs, and there are some cases where we just can't avoid them. They can really be considerable when you're working with large data sets, and if you can solve the problem of data transfer costs from the get-go, you'll be way ahead of many existing services built on S3 and Glacier.
Re: From S3 to R2: An economic opportunity
#60> In fact, there’s an opportunity to build entire companies that take advantage of this price differential and I expect we’ll see more and more of that happening. Interesting. What sort of companies can take advantage of this?