Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

blog.cloudflare.com

Post reply on HN
Print view View original

151–160 of 168 posts

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#151

Post by lolinder »

solardev wrote: post #150
lolinder wrote: post #109

Earlier quoted context omitted.

I found out a few months ago that my router has a built-in VPN that I just had to turn on. Now it doesn't matter where in the world I am, it can always look like I'm not only in the US, I'm actually on my home network. That might be worth considering if you're traveling a lot: it's about as benign-looking to providers as is possible.

I'd be afraid of doing that, knowing how insecure most home routers are. Seems like a zero day (or really even a "zero year", considering how behind updates can be) away from having your home router become part of a botnet...

It's just OpenVPN, and the version was up to date when I enabled it.

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#152

Post by Forbo »

NicoJuicy wrote: post #17
pzmarzly wrote: post #4

It's a seemingly simple and obvious way to lazily migrate your data, but if using Sippy means one less thing for the application code to worry about, and (I assume) is a free add-on, then it provides a ton of value. I have to admit that Cloudflare has been killing it recently with DevX / OpsX. If I wasn't against that company's role in modern internet (as a user of Tor, their firewall is annoying to no end), I would…

Cloudflare doesn't block TOR specifically. It's due to their users and associated behavior that tor Exit nodes have an elevated bad reputation. > https://developers.cloudflare.com/support/firewall/learn-mor... > Due to the behavior of some individuals using the Tor network (spammers, distributors of malware, attackers, etc.), the IP addresses of Tor exit nodes may earn a bad reputation, elevating their Cloudflare thr…

Whatever happened to Privacy Pass? I seem to recall it being touted as the solution for maintaining privacy but keeping bot traffic out, but I haven't heard anything about it in probably five years or so.

I'm also not understanding how enabling Tor routing prevents bot traffic from hitting the site. The traffic gets served over a .onion instead, cool. But how does that prevent the bots?

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#153

Post by uw_rob »

HatchedLake721 wrote: post #130
ludjer wrote: post #77

Earlier quoted context omitted.

Turn on weekly manifest to get a list of files and then you can delete based off that. Much faster and better then listing billions of files.

What’s a weekly manifest?

I believe ludjer is referring to S3 Storage Inventory. This is a daily, or weekly, file produced containing metadata on every file within a S3 Bucket. It does not use the synchronous List APIs.

> You can use Amazon S3 Inventory to help manage your storage. For example, you can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. You can also simplify and speed up business workflows and big data jobs by using Amazon S3 Inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket.

>

> Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#154

Post by abcdefg12 »

kmeisthax wrote: post #110
judge2020 wrote: post #81

Earlier quoted context omitted.

Sure, if you're a business that has to burn money on servers anyways, but a lot of people avoid AWS for side-projects and non-business applications because it can easily balloon to over $100/month with just a few services and moderate traffic.

Also, CloudFront can't proxy bare EC2 servers, you have to shove them behind an ELB, which is additional cost.

Of course it can.

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#155

Post by intellix »

The whole Firewall and rate limiting of CF is amazing and I like where they're going with the workers and Pages. I've also got a lot out our their free plan for many years but now we're on enterprise it feels a bit extortionate. Every email is another contract to sign and another 2k/mo cause we're going over the initial estimation. Much prefer the PAYG model from the likes of AWS and Google etc, even if it were more expensive, it feels more honest

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#157

Post by solardev »

theamk wrote: post #84
solardev wrote: post #64

Earlier quoted context omitted.

The geographical blocks are not enforced by Cloudflare as a blanket ban, but are chosen by each account owner (it's a setting you configure). I've worked with a few companies that saw this as a very valuable service (like small domestic companies blocking international traffic, especially from Russia and China, because we had no presence there anyway and that cut down bot traffic by like 95%). Likewise, TOR access is…

Leaded gasoline was also popular and provided a valuable service to car ownwrs (more power, less engine wear). And yet we banned it. It is not just about customers, you have to thibk about ecosystem as a whole.

What do you mean? You want to force every business to sell internationally?

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#159

Post by tacoship »

judge2020 wrote: post #79
tacoship wrote: post #57

Can I use Cloudflare R2 and Cloudflare CDN to serve HLS video segments? I always hear about ToS issues with Cloudflare and video. I can never seem to find a straight answer.

They created this document for those exclusively using CF services for hosting content including Video: https://www.cloudflare.com/service-specific-terms-developer-... In particular the following line should enable it: > Unlike most Cloudflare products, the Developer Platform can be used to host content. Also see https://blog.cloudflare.com/updated-tos/ > Over time, Cloudflare’s network became larger and more robust…

This really cleared things up for me, thank you!

Re: Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

#160

Post by kmeisthax »

abcdefg12 wrote: post #154
kmeisthax wrote: post #110

Earlier quoted context omitted.

Also, CloudFront can't proxy bare EC2 servers, you have to shove them behind an ELB, which is additional cost.

Of course it can.

So, the CloudFront setup process only surfaces S3, ELBs, API Gateways, Mediastore, and Mediapackage domains as origin domains. I do notice that it will let me type in an arbitrary domain - is that how you're supposed to stick bare EC2 instances behind CloudFront? Just provide it something like realoriginserverplsdonthack.example.com and use some other method (e.g. VPC configuration) to prevent bypassing CloudFront?
Post reply on HN