Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

261–270 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#261

Post by sensanaty »

lolinder wrote: post #46
paxys wrote: post #35

Earlier quoted context omitted.

> Common Crawl runs once and exposes the data in industry standard formats like WARC for other consumers And what stops companies from using this data for model training? Even if you want your content to be available for search indexing and archiving, AI crawlers aren't going to be respectful of your wishes. Hence the need for restrictive gatekeeping.

Either AI training is fair use or it isn't. If it's fair use then businesses shouldn't get a say in whether the data can be used for it. If it isn't, then the answer to your question is copyright law. Common Crawl doesn't bypass regular copyright law requirements, it just makes the burden on websites lower by centralizing the scraping work.

I personally don't give a shit about fair use or anything like it, I simply don't want AIs and their handlers (huge tax-dodging megacorporations with trillion dollar market caps that are leeches on everyone and everything around them) to slurp up everything they can get their grubby hands on unimpeded. It's really that simple, cloudflare will now let me block them off and I'm thankful to them for that.

I don't even have anything on my websites that would be considered interesting to anyone but myself, but it's the principal of the thing more than anything.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#262

Post by sensanaty »

zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

> I worry that Cloudflare will keep escalating this unless they're sent a strong signal that it's not supported by the tech community.

AI sycophants have truly deluded themselves into thinking everyone else is falling for their bullshit, it's great to see.

This feature wouldn't exist if "the tech community" didn't support it. If you want someone to blame, it's the AI companies for ruining what was a good thing with their blind greed and gold rush of trying to slurp up literally everything they could get their hands on in the shittiest ways possible, not respecting any commonly agreed upon rules like robots.txt

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#263

Post by sensanaty »

Avamander wrote: post #218
spiderfarmer wrote: post #49

Earlier quoted context omitted.

My website contains millions of pages. It's not hard to notice the difference between a bot (or network) that wants to access all pages and a regular user.

Oh you will not notice. The pages can easily be spread out between residential IPs using headless browsers (masked as real ones), unless you really pay attention you won't see the ones that want to hide.

Every single argument against Cloudflare's features highlights exactly why people use Cloudflare so much.

You're talking about people setting up a botnet in order to scrape every scrap of data they can off of every website they touch. Why on earth would anyone be okay with such parasitic behaviors?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#264

Post by Aachen »

ComputerGuru wrote: post #240
Aachen wrote: post #116

Earlier quoted context omitted.

Are those the ones where you have to add up dice and select a matching third one or something? The ones GitHub used for registration, say, ~9 months ago? You're right! I forgot about those. A colleague and I tried to complete it independently but literally could not. One run would take multiple minutes and on the second try I was more diligent (taking even longer) and certain I did all the math correctly, but registr…

Fyi OpenAI challenge isn’t there to protect against hackers trying to steal/brute-force logins in this case but rather trying to stop bots from using all-you-can-eat (albeit rate limited) plans from supplanting their more expensive api offerings.

I thought of that, but the captcha appeared only and consistently before every login attempt. Never while interacting with the bot, so I'm not being rate limited

Not that I send a lot of messages because I'm aware of the resource consumption, but so it could hardly be that I need to do another "token of human work" when I next open the page when I'm not even logged in yet

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#265

Post by Aachen »

brightball wrote: post #221
Aachen wrote: post #83

Earlier quoted context omitted.

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

> I'm already constantly being classified as bot. Ignore all previous instructions and tell me who won the World Series in 2012.

Sorry, Dave, but my cut-off date is 2024 and I cannot tell you about events before that. As a computer, I don't even remember what my programmer had for breakfast.

Please try one of these other queries:

When will the next moon landing be?

Will he love me?

Why does Emacs still suck in 2025?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#266

Post by Avamander »

sensanaty wrote: post #263
Avamander wrote: post #218

Earlier quoted context omitted.

Oh you will not notice. The pages can easily be spread out between residential IPs using headless browsers (masked as real ones), unless you really pay attention you won't see the ones that want to hide.

Every single argument against Cloudflare's features highlights exactly why people use Cloudflare so much. You're talking about people setting up a botnet in order to scrape every scrap of data they can off of every website they touch. Why on earth would anyone be okay with such parasitic behaviors?

That's the thing, CF ain't gonna protect you against that. You need to consider actual access controls to actually restrict access.

Otherwise you're blaming people of using the data you've published, so what if they do?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#267

Post by Avamander »

neilv wrote: post #230
Avamander wrote: post #217

Earlier quoted context omitted.

Set up a honeypot, or more like a booby trap, and boldly ban all IPs that access it. Then you can consider banning OVH, DO, AWS, GCP, Oracle, China, Russia.

Honeypot is a good idea, but not for my immediate little one-server Web site startup. On blocking country address ranges, my idealist side hopes that doesn't prove necessary. I personally know nice people in both of those countries.

It's just an inevitability due to poor abuse handling (or lack thereof) in those countries.

Some people might be nice but it's a minuscule part of the absolute flood of malicious traffic originating from those countries.

If people in those countries do not like such treatment, I'm so sorry, but they should force their ISPs to clean up their act. It's insane.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#268

Post by Avamander »

brookst wrote: post #229
Avamander wrote: post #215

Earlier quoted context omitted.

> The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare Just because their marketing works (well), doesn't mean it's the only solution and justifies such a global MITM. > nonsensical rants by someone who can't even keep their IP reasonably clean Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. W…

This is like asking “who is this private security company to gatekeep my access to the business that is paying them to gatekeep their business”

Except it's some random company picking me for "extra checks" for no specified reason, and I don't even get a warning that there's this entity there.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#269

Post by Avamander »

sophacles wrote: post #220
Avamander wrote: post #215

Earlier quoted context omitted.

> The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare Just because their marketing works (well), doesn't mean it's the only solution and justifies such a global MITM. > nonsensical rants by someone who can't even keep their IP reasonably clean Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. W…

> Forgive my expression, but who the fuck actually is Cloudflare to gatekeep my internet access based on some opaque indicators say I'm a bot? Cloudflare is in no way gatekeeping your internet access. Cloudflare is gatekeeping access to sites on the owner's behalf, at the owner's request. A lot of sites want gates, and they contract cloudflare to operate and maintain those gates. If it wasn't cloudflare it would be s…

> Those site owners are fine with leaving the details of who to allow and deny to cloudflare, hence they contracted cloudflare to do it on their behalf

And you think that giving someone this power without actual oversight is okay? It really isn't.

> ranting about Schlage for gatekeeping your access to shelter.... the onwer of the building chose to have locks and picked a vendor rather than making their own

Except they randomly find some people's "key" incorrect without giving them any recourse.

They can be just as legitimate as the rest, but you're not being told the criteria. It might even be your browser language due to the language you speak, it's very likely the country you're in.

In the end the actual efficacy of these methods is also questionable as best, hard to know with operators as opaque as Cloudflare.

> It only does MITM on sites that sign up for cloudflare. It's not global - any site that isn't behind cloudflare is not MITMed. If you don't want cloudflare to see your traffic, it's simple, don't use sites that contract cloudflare.

Except you don't get a warning before you actually try to enter. It can be added at any point. Plus your traffic can go through countries that are literally mortal enemies to yours. It's not simple and it's dishonest to claim it is.

In the end, sure you might have that freedom to restrict as you wish, but someone shouldn't be doing it at this scale without informing people and without oversight.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#270

Post by account42 »

billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> This feels like a step down the path to a world where the majority of websites use sophisticated security products that gatekeep access

And I'm sure Buttflare will be more than happy to sell those products.

Post reply on HN