Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

51–60 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#51

Post by red_admiral »

zkid18 wrote: post #45

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

The website owner chooses. They can say "nope" in robots.txt. Not everyone respects this, but Google does. Google can choose not to show that site as a result, if they want to.

This adds a third option besides yes and no, which is "here's my price". Also, because cloudflare is involved, bots that just ignore a "nope" might find their lives a bit harder.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#52

Post by lolinder »

zkid18 wrote: post #45

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

Yeah, there's a lot of confusion between AI training and AI agent access, and it's dangerous.

Training embeds the data into the model and has copyright implications that aren't yet fully resolved. But an AI agent using a website to do something for a user is not substantially different than any other application doing the same. Why does it matter to you, the company, if I use a local LLaMA to process your website vs an algorithm I wrote by hand? And if there is no difference, are we really comfortable saying that website owners get a say in what kinds of algorithms a user can run to preprocess their content?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#53

Post by l5870uoo9y »

spiderfarmer wrote: post #49
creatonez wrote: post #15

This seems like a gimmick. Isn't preventing crawling a sisyphean task? The only real difference this will make is further entrenching big players who have already crawled a ton of data. And if this feature comes at the cost of false positives and overbearing captchas, it will start to affect users.

My website contains millions of pages. It's not hard to notice the difference between a bot (or network) that wants to access all pages and a regular user.

How often are the bots indexing it?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#54

Post by lolinder »

red_admiral wrote: post #51
zkid18 wrote: post #45

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

The website owner chooses. They can say "nope" in robots.txt. Not everyone respects this, but Google does. Google can choose not to show that site as a result, if they want to. This adds a third option besides yes and no, which is "here's my price". Also, because cloudflare is involved, bots that just ignore a "nope" might find their lives a bit harder.

Robots.txt is for crawlers. It's explicitly not meant to say one-off requests from user agents can't access the site, because that would break the open web.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#55

Post by jsheard »

lolinder wrote: post #52
zkid18 wrote: post #45

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

Yeah, there's a lot of confusion between AI training and AI agent access, and it's dangerous. Training embeds the data into the model and has copyright implications that aren't yet fully resolved. But an AI agent using a website to do something for a user is not substantially different than any other application doing the same. Why does it matter to you, the company, if I use a local LLaMA to process your website vs…

> But an AI agent using a website to do something for a user is not substantially different than any other application doing the same.

If the website is ad-supported then it is substantially different - one produces ad impressions and the other doesn't. Adblocking isn't unique to AI agents of course but I can see why site owners wouldn't want to normalize a new means of accessing their content which will inherently never give them any revenue in return.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#56

Post by lolinder »

jsheard wrote: post #55
lolinder wrote: post #52

Earlier quoted context omitted.

Yeah, there's a lot of confusion between AI training and AI agent access, and it's dangerous. Training embeds the data into the model and has copyright implications that aren't yet fully resolved. But an AI agent using a website to do something for a user is not substantially different than any other application doing the same. Why does it matter to you, the company, if I use a local LLaMA to process your website vs…

> But an AI agent using a website to do something for a user is not substantially different than any other application doing the same. If the website is ad-supported then it is substantially different - one produces ad impressions and the other doesn't. Adblocking isn't unique to AI agents of course but I can see why site owners wouldn't want to normalize a new means of accessing their content which will inherently n…

I don't believe that companies have the right to say that my user agent must run their ads. They can politely request that it does and I can tell my agent whether to show them or not.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#57

Post by brigadier132 »

zkid18 wrote: post #45

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

For traditional search indexing the interests of the aggregator and the content creator were aligned. AIs on the other hand are adversarial to the interest of content creators, a sufficiently advanced AI can replace the creator of the content it was trained on.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#58

Post by spacebanana7 »

creatonez wrote: post #15

This seems like a gimmick. Isn't preventing crawling a sisyphean task? The only real difference this will make is further entrenching big players who have already crawled a ton of data. And if this feature comes at the cost of false positives and overbearing captchas, it will start to affect users.

> The only real difference this will make is further entrenching big players

It's the opposite. Only big players like google get meetings with big publishers and copyright holders to be individually whitelisted in robots.txt. Whereas a marketplace is accessible to any startup or university.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#59

Post by jsheard »

lolinder wrote: post #56
jsheard wrote: post #55

Earlier quoted context omitted.

> But an AI agent using a website to do something for a user is not substantially different than any other application doing the same. If the website is ad-supported then it is substantially different - one produces ad impressions and the other doesn't. Adblocking isn't unique to AI agents of course but I can see why site owners wouldn't want to normalize a new means of accessing their content which will inherently n…

I don't believe that companies have the right to say that my user agent must run their ads. They can politely request that it does and I can tell my agent whether to show them or not.

True, but by the same measure your user agent can politely request a webpage and the server has the right to say 403 Forbidden. Nobody is required to play by the other parties rules here.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#60

Post by 1986 »

brigadier132 wrote: post #41
neilv wrote: post #37

Earlier quoted context omitted.

> This kind of cynicism is boring. IMHO, this kind of thinking is only cynicism iff you're only looking for your angle to profit, and someone is peeing on your parade, every time they boorishly mention irrelevant, imaginary concerns like "ethics", "legality", or "Geneva Convention".

[flagged]

Interesting, as my theory for why cynicism is so common nowadays is that it's a coping mechanism for people who understand perfectly well what's happening in the modern world
Post reply on HN