Live data from Hacker News

Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

blog.cloudflare.com

71–80 of 168 posts

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#71
post #3

This is amazing. I’m thinking of building the same thing for self hosted S3 (ie. Minio). I love AWS, they're very reliable and i feel like depending on them is not risky. But being 100% dependent on a single provider scares me a lot, and I often feel like S3 is the one thing that would cost me a lot if I tried to migrate. I have not accounted for all the data I have there, but it's lots of small files (all the user u…

Bulk deletes on S3 should be done using a Life Cycle Policy to avoid the per object delete cost

Deletes are free, but you have to pay to list to delete if not individually tracking.

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#72
post #63

Earlier quoted context omitted.

Well no, cred stuffing often results in termination or suspension of service by the merchant accounts used by the victim. This denies them the ability to sell thier goods and services - generally this denies end users the ability to ise the victim's service just as effectively as a volumetric attack. Further, cred stuffing is often automated by a botnet. The two things are distinct, but have similar means and end res…

And a common thread is that they originate traffic from sleazy IPs: having a service like Cloudflare block those will protect against ton of internet background noise with very little downside for most businesses, so it’s unsurprising that so many sites do so.

I was surprised to find 1000s of google cloud and azure ips last attack.

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#73

Earlier quoted context omitted.

The geographical blocks are not enforced by Cloudflare as a blanket ban, but are chosen by each account owner (it's a setting you configure). I've worked with a few companies that saw this as a very valuable service (like small domestic companies blocking international traffic, especially from Russia and China, because we had no presence there anyway and that cut down bot traffic by like 95%). Likewise, TOR access is…

I am convinced that while harder, there are more intelligent ways to block these DDoS attacks other than blocking entire geographies. I often travel between Africa and US, and there are things like buying furniture (home depot blanket blocks non US customers, but they would simply allow shipping only to US addresses), buying cars (there are large car sites that don't allow browsing from outside US, even if you've alr…

> I am convinced that while harder, there are more intelligent ways to block these DDoS attacks other than blocking entire geographies.

Sure. Even by default, Cloudflare won't block entire countries. That's a CHOICE some businesses make if the default blocks aren't enough, and they don't have the time or resources to configure more nuanced WAF rules. (OWASP isn't exactly straightforward). Edit: For example, at that job I was talking about, we had different rulesets for different regions... China and Russia were completely banned, Africa was put behind stricter JS security checks and CAPTCHAs but allowed in, Europe had a medium security level (we did occasionally sell there, but very rarely), while the US had entirely custom WAF rules. It just depends on who we wanted to sell to or not.

It goes the other way around, too, you know. I've seen European and Asian sites that geoblock US customers. It's not out of malice, they just don't want to deal with the edge cases. Even if a foreign customer can access your website and buy stuff, dealing with international customs, consumer laws, credit card fraud, wire transfers, etc. can be a pain that's not worth it for smaller merchants. And if the foreign buyer is using a reshipper anyway, well, the reshipper can just buy the whole thing for them and deal with payments, etc. as an intermediary, like how Tenso/BuyFromJapan/JapanRabbit work.

Big companies have proper international presences, but for small local businesses, the amount of effort it takes to support international buyers just isn't worth the profit they typically bring in. Even on eBay, with its built-in international payment and shipping rules, sellers often won't want to bother.

This isn't really a matter of security rules, really, but just business cost/benefit decisions.

Besides, it helps businesses in each country stay local! Do you really want Amazon taking over everywhere...?

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#74

Earlier quoted context omitted.

The geographical blocks are not enforced by Cloudflare as a blanket ban, but are chosen by each account owner (it's a setting you configure). I've worked with a few companies that saw this as a very valuable service (like small domestic companies blocking international traffic, especially from Russia and China, because we had no presence there anyway and that cut down bot traffic by like 95%). Likewise, TOR access is…

I am convinced that while harder, there are more intelligent ways to block these DDoS attacks other than blocking entire geographies. I often travel between Africa and US, and there are things like buying furniture (home depot blanket blocks non US customers, but they would simply allow shipping only to US addresses), buying cars (there are large car sites that don't allow browsing from outside US, even if you've alr…

is shipping forwarding common?

i would think that it adds on a huge cost?

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#75
post #4

It's a seemingly simple and obvious way to lazily migrate your data, but if using Sippy means one less thing for the application code to worry about, and (I assume) is a free add-on, then it provides a ton of value. I have to admit that Cloudflare has been killing it recently with DevX / OpsX. If I wasn't against that company's role in modern internet (as a user of Tor, their firewall is annoying to no end), I would…

What's the alternative if most of tor traffic is password attempts and bad actors how do you protect yourself from tors bad actors without effecting all of tors users. I work at a company that runs a large website top 1000 websites in the world, and we don't even have to block tor exit nodes since they trigger our bot and snap blocking rules on our firewall, how do we let valid for users through without letting all the malicious actors?

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#77

Earlier quoted context omitted.

Bulk deletes on S3 should be done using a Life Cycle Policy to avoid the per object delete cost

Deletes are free, but you have to pay to list to delete if not individually tracking.

Turn on weekly manifest to get a list of files and then you can delete based off that. Much faster and better then listing billions of files.

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#78
post #4

It's a seemingly simple and obvious way to lazily migrate your data, but if using Sippy means one less thing for the application code to worry about, and (I assume) is a free add-on, then it provides a ton of value. I have to admit that Cloudflare has been killing it recently with DevX / OpsX. If I wasn't against that company's role in modern internet (as a user of Tor, their firewall is annoying to no end), I would…

Cloudflare doesn't hate Tor. Blocking Tor is purely a customer configuration

>Blocking Tor is purely a customer configuration

...that's on by default and so used by the vast majority of Cloudflare customers making it effectively a Cloudflare configuration.

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#79

Can I use Cloudflare R2 and Cloudflare CDN to serve HLS video segments? I always hear about ToS issues with Cloudflare and video. I can never seem to find a straight answer.

They created this document for those exclusively using CF services for hosting content including Video: https://www.cloudflare.com/service-specific-terms-developer-...

In particular the following line should enable it:

> Unlike most Cloudflare products, the Developer Platform can be used to host content.

Also see https://blog.cloudflare.com/updated-tos/

> Over time, Cloudflare’s network became larger and more robust and its portfolio broadened to include services like Stream, Images, and R2. These services are explicitly designed to allow customers to serve non-HTML content like video, images, and other large files hosted directly by Cloudflare.

> Video and large files hosted outside of Cloudflare will still be restricted on our CDN, but we think that our service features, generous free tier, and competitive pricing (including zero egress fees on R2) make for a compelling package for developers that want to access the reach and performance of our network.

(of course this is not legal advice)

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#80

9 cents a gigabyte downloaded versus 0 cents a gigabyte downloaded is a pretty good deal. There’s not much AWS can do about it because they must make untold billions from those sweet, sweet S3 egress fees. I’d be willing to bet S3 egress fees make up about 60% of all AWS revenue.

Anyone doing serious traffic with AWS will use cloud front and then you get additional savings on-top of that and you can negotiate a better rate then provided depending on usage so some people pay way less then the advertised price.
Post reply on HN