How would you do the opposite of this? Optimize your content to be more likely crawled by AI bots? I know traditional Google-focused SEO is not enough because these AI bots often use other web search/indexing APIs.
Cloudflare Introduces Default Blocking of A.I. Data Scrapers
61–70 of 342 posts
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#62I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…
I wonder… Google scrapes for indexing and for AI, right? I wonder if they will eventually say: ok, you can have me or not, if you don’t want to help train my AI you won’t get my searches either. That’s a tough deal but it is sort of self-consistent.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#63I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…
Not much into that kind of DevOps. What is a good basic caching in this instance?
Cloudflare and other CDNs will usually automatically cache your static pages.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#64How would you do the opposite of this? Optimize your content to be more likely crawled by AI bots? I know traditional Google-focused SEO is not enough because these AI bots often use other web search/indexing APIs.
There are script tags you can put in your site from LLM SEO companies if you want your content to be indexed by Perplexity or OpenAI. Theyre kind of too new for me to reccomend.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#65I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…
The presumption I’m already using cloudfare is a start. Is this a requirement for maintaining a simple website now?
Either that or Anubis (https://anubis.techaro.lol/docs), yes.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#66The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare. The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.
They cant do anything other than bog down the internet. I havent found a single cf provided challenge I havent been able to get past in This is simply juat the first step in them implementing a marketplace and trying to get into LLM SEO. They dont care about your site or protecting it. They are gearing up to start making a cut in the Middle between scrapers and publishers. Why wouldnt I go DIRECTLY to the publisher and make a deal. So dumb I hate cf so much.
The only thing cloudflare knows how to do is MITM attacks.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#67The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare. The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.
It is now an adversarial relationship between aibots and website, and cloudflare is merely reacting to it. Would you say the same for ddos protection? Isn't that the same as well?
They arent doing anything. They are attempting to insert themselves into the middle of a marketplace (that doesnt exist and never will) where scrapers pay for IP. They think theyre going to profit off the bots, not protect your site. Dont fall for their scam.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#68Earlier quoted context omitted.
With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Goo…
I don't think we should ban ad blockers, but I also think it's fair to suggest that the loss of organic traffic could be affecting the incentive to create new digital content, at least as much as the fear of having your content absorbed into an LLM's training data.
IMO the backlash against LLMs is more philosophical, a lot of people don’t like them or the idea of one learning from their content. Unless your website has some unique niche information unavailable anywhere else there’s no direct personal risk. RAG would be a more direct threat if anything.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#69The list of bots is pretty short right now: https://developers.cloudflare.com/bots/concepts/bot/#ai-bots
Cloudflare sees a lot of the web traffic. I assume these are the biggest bots they're seeing right now, and any new contenders would be added as they find them. Probably impossible to really block everything, but they've got the web-coverage to detect more than most.
They are lying. They cant detect crawlers unless we tell them we are who we are.
Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers
#70Earlier quoted context omitted.
This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This…
With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Goo…
What I want to know is if the flood of scraping everyone has been complaining about is coming from people trying to scrape for training or bots doing RAG search.
I get that everyone wants data, but presumably the big players already scraped the web. Do they really need to do it again? Or is it bit players reproducing data that's likely already in the training set? Or is it really that valuable to have your own scraped copy of internet scale data?
I feel like I'm missing something here. My expectation is that RAG traffic is going to be orders of magnitude higher than scraping for training. Not that it would be easy to measure from the outside.