Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

31–40 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#31

Post by loceng »

neilv wrote: post #11

Cloudflare found a new variation on their traditional service of protecting from abusers. This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers. And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment. I'd need to ask the lawyer how close this is to technically being a protection rack…

If they don't offer to just block the bots instead of you signing on, then I imagine it'd easily be seen as a racket.

How much effort then Cloudflare puts on tracking circumvention efforts of bot networks is then another question.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#33

Post by jsheard »

neilv wrote: post #11

Cloudflare found a new variation on their traditional service of protecting from abusers. This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers. And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment. I'd need to ask the lawyer how close this is to technically being a protection rack…

> I'd need to ask the lawyer how close this is to technically being a protection racket, or other no-no.

Wait 'til you find out how many of the DDoS-for-hire services that Cloudflare offers to protect you from are themselves protected by Cloudflare.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#34

Post by matt-p »

jsheard wrote: post #17
tomjen3 wrote: post #12

Earlier quoted context omitted.

This won't work. If you are doing an AI startup, you will want to use GoogleBot for your crawler and this will bypass that. Not too much of a loss, since the only quality content is already behind paywalls, or on diverse wikistyle sites. Anything served with ads for commercial reasons is automatically drivel, based on my experience. There simply isn't a business in making it better. Edit: updated comment to not be ne…

It is trivial to detect fake GoogleBot traffic (Google provides ways to validate it) and Cloudflare already does so. See for yourself: curl -I -H "User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/105.0.5195.102 Safari/537.36" https://www.cloudflare.com They'll immediately flag the request as malicious and return 403 Forbidden, even if yo…

Now try it from a google cloud vm.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#35

Post by paxys »

billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> Common Crawl runs once and exposes the data in industry standard formats like WARC for other consumers

And what stops companies from using this data for model training? Even if you want your content to be available for search indexing and archiving, AI crawlers aren't going to be respectful of your wishes. Hence the need for restrictive gatekeeping.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#36

Post by jsheard »

matt-p wrote: post #34
jsheard wrote: post #17

Earlier quoted context omitted.

It is trivial to detect fake GoogleBot traffic (Google provides ways to validate it) and Cloudflare already does so. See for yourself: curl -I -H "User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/105.0.5195.102 Safari/537.36" https://www.cloudflare.com They'll immediately flag the request as malicious and return 403 Forbidden, even if yo…

Now try it from a google cloud vm.

Pretty sure that won't work, they let you validate whether an IP address is used by GoogleBot specifically, not just owned by Google in general. I doubt they are foolish enough to use the same pool of IP addresses for their internal crawlers and their public cloud.

https://developers.google.com/search/docs/crawling-indexing/...

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#37

Post by neilv »

brigadier132 wrote: post #30
neilv wrote: post #11

Cloudflare found a new variation on their traditional service of protecting from abusers. This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers. And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment. I'd need to ask the lawyer how close this is to technically being a protection rack…

[flagged]

> This kind of cynicism is boring.

IMHO, this kind of thinking is only cynicism iff you're only looking for your angle to profit, and someone is peeing on your parade, every time they boorishly mention irrelevant, imaginary concerns like "ethics", "legality", or "Geneva Convention".

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#38

Post by giancarlostoro »

nindalf wrote: post #25
giancarlostoro wrote: post #8

I really love Cloudflare. They're always up to something interesting and different. I hope we see more companies rise up similar to Cloudflare. I almost want to say Cloudflare is everything we hoped Google would be, but Google became another corporate cog machine that innovates and then scraps things up in one swoop. I don't recall the last I heard of Cloudflare spinning something up just to wind it back down? I don'…

> last I heard of Cloudflare spinning something up just to wind it back down Cloudflare bet big on NFTs ( https://blog.cloudflare.com/cloudflare-stream-now-supports-n... ), Web3 ( https://blog.cloudflare.com/get-started-web3/ ), Proof of stake ( https://blog.cloudflare.com/next-gen-web3-network/ ). In fact they "bet on blockchain" way back in 2017 ( https://blog.cloudflare.com/betting-on-blockchain/ ) but it's tellin…

Im neutral on crypto, I see it like AI, its just waiting on some breakthrough that pulls everyone. My suspicion is someone needs to make it stupid easy to get into crypto.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#39

Post by billyhoffman »

paxys wrote: post #35
billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> Common Crawl runs once and exposes the data in industry standard formats like WARC for other consumers And what stops companies from using this data for model training? Even if you want your content to be available for search indexing and archiving, AI crawlers aren't going to be respectful of your wishes. Hence the need for restrictive gatekeeping.

Licensing. Common Crawl could change the license of how the data it produces is used.

Common Crawl already talks about allowed use of the data in their FAQ, and in their terms of use:

https://commoncrawl.org/terms-of-use/ https://commoncrawl.org/faq

While this doesn't currently discuss AI, they could. This would allow non-AI downstream consumers to not be penalized.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#40

Post by matt-p »

jsheard wrote: post #36
matt-p wrote: post #34

Earlier quoted context omitted.

Now try it from a google cloud vm.

Pretty sure that won't work, they let you validate whether an IP address is used by GoogleBot specifically, not just owned by Google in general. I doubt they are foolish enough to use the same pool of IP addresses for their internal crawlers and their public cloud. https://developers.google.com/search/docs/crawling-indexing/...

It depends how the site has implemented it, a huge number just look for AS origination and *googleuserconent.com
Post reply on HN