Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

1–10 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#4

Post by hedora »

boristsr wrote: post #2

I'm pretty interested in how companies are exploring how to properly monetize or compensate for scraped content to help keep a strong ecosystem of quality content. Id love to see more efforts like this.

Companies have been trying to find novel ways to bypass fair use / public domain laws for a long time.

Each time they do, we see more consolidation of the media, and lower pay for the people that produce the content.

I don’t see why this particular effort will turn out differently.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#5

Post by kordlessagain »

boristsr wrote: post #2

I'm pretty interested in how companies are exploring how to properly monetize or compensate for scraped content to help keep a strong ecosystem of quality content. Id love to see more efforts like this.

There's a HTTP code for charging for access: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402

Then there's a Lightning Network protocol for it: https://docs.lightning.engineering/the-lightning-network/l40...

With the Cloudflare stuff, it just seems like an excuse to sell Cloudflare services (and continue to force everyone to use it) as opposed to just figuring out a standard way of using what is already built to provide access for some type of micropayment.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#6

Post by neilv »

> A demo of AI Audit shared with TechCrunch showed how website owners can use the tool to see how AI models are scraping their sites. Cloudflare’s tool is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site.

And if I didn't authorize the freeloading copyright-laundering service companies to pound my server and take my content, then I need a really good lawyer, with big teeth and claws.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#7

Post by BSDobelix »

neilv wrote: post #6

> A demo of AI Audit shared with TechCrunch showed how website owners can use the tool to see how AI models are scraping their sites. Cloudflare’s tool is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. And if I didn't authorize the freeloading copyright-launderi…

I would say let's get rid of copyright and software patents altogether ;)

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#8

Post by giancarlostoro »

I really love Cloudflare. They're always up to something interesting and different. I hope we see more companies rise up similar to Cloudflare. I almost want to say Cloudflare is everything we hoped Google would be, but Google became another corporate cog machine that innovates and then scraps things up in one swoop. I don't recall the last I heard of Cloudflare spinning something up just to wind it back down? I don't think its impossible for them to make a bad choice, but I think they really think their projects through typically.

My biggest problem with AI will be once it starts getting legislated, it will just be limited in how it can function / be built, we are going to lock in existing LLMs like ChatGPT in the lead and stop anyone from competing since they wont be able to train on the same data.

My other biggest problem is "AI" or really LLMs which is what everyones hyped about, is lack of offline first capabilities.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#10

Post by jsheard »

kordlessagain wrote: post #5
boristsr wrote: post #2

I'm pretty interested in how companies are exploring how to properly monetize or compensate for scraped content to help keep a strong ecosystem of quality content. Id love to see more efforts like this.

There's a HTTP code for charging for access: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 Then there's a Lightning Network protocol for it: https://docs.lightning.engineering/the-lightning-network/l40... With the Cloudflare stuff, it just seems like an excuse to sell Cloudflare services (and continue to force everyone to use it) as opposed to just figuring out a standard way of using what is already b…

The problem is that soft technical measures like HTTP 402 and robots.txt aren't legally binding, so there's nothing stopping scrapers from just ignoring them. Cloudflares value proposition here is they will play the cat-and-mouse game of detecting things like spoofed user agents and residential proxies on your behalf, and actively block what appears to be scraper traffic unless they pay up.

Unfortunately this probably means even more CAPTCHAs for people using VPNs and other privacy measures as they ramp up the bot detection heuristics.

Post reply on HN