Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

11–20 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#11
Cloudflare found a new variation on their traditional service of protecting from abusers.

This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers.

And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment.

I'd need to ask the lawyer how close this is to technically being a protection racket, or other no-no.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#12
post #2

I'm pretty interested in how companies are exploring how to properly monetize or compensate for scraped content to help keep a strong ecosystem of quality content. Id love to see more efforts like this.

This won't work. If you are doing an AI startup, you will want to use GoogleBot for your crawler and this will bypass that.

Not too much of a loss, since the only quality content is already behind paywalls, or on diverse wikistyle sites. Anything served with ads for commercial reasons is automatically drivel, based on my experience. There simply isn't a business in making it better.

Edit: updated comment to not be needlessly diversive.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#13

I really love Cloudflare. They're always up to something interesting and different. I hope we see more companies rise up similar to Cloudflare. I almost want to say Cloudflare is everything we hoped Google would be, but Google became another corporate cog machine that innovates and then scraps things up in one swoop. I don't recall the last I heard of Cloudflare spinning something up just to wind it back down? I don'…

Someone somewhere outside of your country's legal entities can still do all the things your country doesn't like and there's little to stop them. Governments might limit legal or commercial usage but it doesn't mean it won't exist.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#15
This seems like a gimmick. Isn't preventing crawling a sisyphean task? The only real difference this will make is further entrenching big players who have already crawled a ton of data. And if this feature comes at the cost of false positives and overbearing captchas, it will start to affect users.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#16
post #2

I'm pretty interested in how companies are exploring how to properly monetize or compensate for scraped content to help keep a strong ecosystem of quality content. Id love to see more efforts like this.

> help keep a strong ecosystem of quality content

To the extent quality content does exist online: what isn't either already behind a paywall, or created by someone other than who will be compensated under such a scheme?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#17
post #12
post #2

I'm pretty interested in how companies are exploring how to properly monetize or compensate for scraped content to help keep a strong ecosystem of quality content. Id love to see more efforts like this.

This won't work. If you are doing an AI startup, you will want to use GoogleBot for your crawler and this will bypass that. Not too much of a loss, since the only quality content is already behind paywalls, or on diverse wikistyle sites. Anything served with ads for commercial reasons is automatically drivel, based on my experience. There simply isn't a business in making it better. Edit: updated comment to not be ne…

It is trivial to detect fake GoogleBot traffic (Google provides ways to validate it) and Cloudflare already does so. See for yourself:

  curl -I -H "User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/105.0.5195.102 Safari/537.36" https://www.cloudflare.com
They'll immediately flag the request as malicious and return 403 Forbidden, even if your IP address is otherwise reputable.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#18
post #6

> A demo of AI Audit shared with TechCrunch showed how website owners can use the tool to see how AI models are scraping their sites. Cloudflare’s tool is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. And if I didn't authorize the freeloading copyright-launderi…

I would say let's get rid of copyright and software patents altogether ;)

they're already gone

but only if you're well funded (OpenAI)

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#19
post #13

I really love Cloudflare. They're always up to something interesting and different. I hope we see more companies rise up similar to Cloudflare. I almost want to say Cloudflare is everything we hoped Google would be, but Google became another corporate cog machine that innovates and then scraps things up in one swoop. I don't recall the last I heard of Cloudflare spinning something up just to wind it back down? I don'…

Someone somewhere outside of your country's legal entities can still do all the things your country doesn't like and there's little to stop them. Governments might limit legal or commercial usage but it doesn't mean it won't exist.

Its much harder to pull off when you're hitting an international market, are you really going to ignore an entire country? Maybe if it was a small country with few citizens, but if the EU or US passes a law, you're going to miss out on an entire market.
Post reply on HN