Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare to introduce pay-per-crawl for AI bots

blog.cloudflare.com

Post reply on HN
Print view View original

81–90 of 317 posts

Re: Cloudflare to introduce pay-per-crawl for AI bots

#81

Post by asimpletune »

It’s a step in the right direction but I think there’s a long ways to go. Even better would be pay-for-usage. So if you want to crawl a site for research, then it should be practically free, for example. If you want to crawl a site to train a bot that will be sold then it should cost a lot.

I am truly sorry to even be thinking along these lines, but the alternative mindset has been made practically illegal in the modern world. I would 100% be fine with there being a world library that strives to provide access to any and all information for free, while also aiming to find a fair way to compensate ip owners… technology has removed most of the technical limitations to making this a reality AND I think the net benefit to humanity would be vastly superior to the cartel approach we see today.

For now though that door is closed so instead pay me.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#82

Post by danaris »

asimpletune wrote: post #81

It’s a step in the right direction but I think there’s a long ways to go. Even better would be pay-for-usage. So if you want to crawl a site for research, then it should be practically free, for example. If you want to crawl a site to train a bot that will be sold then it should cost a lot. I am truly sorry to even be thinking along these lines, but the alternative mindset has been made practically illegal in the mod…

The problem with this is that people who want to make money will always be highly motivated to either find loopholes to abuse the system, outright lie about their intentions, buy and resell the data for less (making profit on volume), or just break in.

"Ah, it's free for research? Well, that's what I'm doing! I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to this company that is coincidentally also owned by me to sell it!"

Re: Cloudflare to introduce pay-per-crawl for AI bots

#83

Post by JimDabell »

marginalia_nu wrote: post #74
JimDabell wrote: post #33

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost. Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastru…

Well there's common crawl, which is supposed to be that. Though ironically it's been under so much load from AI startups attempting to greedily gobble down its data it was basically inaccessible the last time I tried to use it. Turtles all the way down it seems. There's probably a gap in the market for something like this. Crawling is a bit of a hassle and being able to outsource it would help a lot of companies. Not…

Common Crawl is great, but it only updates monthly and doesn’t do transformations. It’s good for seeding a search engine index initially, but wouldn’t be suitable for ongoing use. But it’s generally the kind of thing I’m talking about, yeah.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#84

Post by Melonai »

Leynos wrote: post #43
suyash wrote: post #5

Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.

Is this companies collecting data for model training, or is it agentic tools operating on behalf of users?

I think in the grand scheme of things barely anyone uses agents (as of now) to crawl sites quickly apart from maybe a quick Google search or two. At least that's been my observation of my non-technical field friends using LLMs.

From what it looks like in the web logs it is in fact the same few AI company web crawlers constantly crawling and recrawling the same URLs over and over, presumably to get even the slightest advantage over each other, as they are definitely in the zero-sum mindset currently.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#87

Post by arccy »

some_furry wrote: post #17
aspenmayer wrote: post #15

Earlier quoted context omitted.

I mean, I wouldn't care who referred the paying user. I would optimize my blog to serve free and paying users to the best of my ability. I would hope that I could do that but I don't know what you are trying to do with your blog. Most blogs could probably be self-hosted with some caching and static layout where possible to perhaps avoid needing to use Cloudflare. I guess you already have to be using CF to have access…

I pay $300/year for the privilege of being able to write what I want without the pressure to monetize or surveil my readers. Some of my blog posts are linked by programming language docs for cryptography. Others have helped queer folks transition to a higher-paying tech career. It's difficult to quantify either of those things in a dollar value. I've opted to not ever do so. But if I can make the AI slop machines pay…

that $300 sounds more like a personal choice when you can do it for ~free, e.g. with GitHub pages or similar.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#88

Post by yen223 »

mhandley wrote: post #85

That sounds reasonable for access to actual content, but it produces a huge new incentive to constantly produce vast amounts of AI-generated slop served via Cloudflare. Is there a way to disincentivize this?

I presume the onus will now be on the AI scrapers to decide whether that AI-slop site is worth paying for. How they will figure this out will be interesting to see.

Re: Cloudflare to introduce pay-per-crawl for AI bots

#89

Post by samrus »

FloatArtifact wrote: post #64

What about if somebody uses artificial intelligence crawler to help them navigate the web as an accessibility tool? Enabling UI automation. It already throws up a lot of... uh... troublesome verifications.

The site owner can allow such crawlers. There is the issue of bad actors pretending to be these types of crawlers but that could already happen to a site that want to allow google search crawlers but not gemini training data crawlers for example, so theres strong support to solve that problem

Re: Cloudflare to introduce pay-per-crawl for AI bots

#90

Post by joosters »

danaris wrote: post #82
asimpletune wrote: post #81

It’s a step in the right direction but I think there’s a long ways to go. Even better would be pay-for-usage. So if you want to crawl a site for research, then it should be practically free, for example. If you want to crawl a site to train a bot that will be sold then it should cost a lot. I am truly sorry to even be thinking along these lines, but the alternative mindset has been made practically illegal in the mod…

The problem with this is that people who want to make money will always be highly motivated to either find loopholes to abuse the system, outright lie about their intentions, buy and resell the data for less (making profit on volume), or just break in. "Ah, it's free for research? Well, that's what I'm doing! I'm conducting research! Ignore the fact that once I have the data, I'm going to turn around and give it to t…

You can tell the difference between the two by checking if the Evil bit is set in the corresponding IP packet - RFC 3514 already standardised this.
Post reply on HN