Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

71–80 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#71

Post by TechDebtDevin »

chasd00 wrote: post #20
Sol- wrote: post #9

Do the major AI companies actually honor robots.txt? Even if some of their publicly known crawlers might do it, surely they have surreptitious campaigns where they do some hidden crawling, just like how they illegally pirate books, images and user data to train on.

My thought too, honoring robots.txt is just a convention. There's no requirement to follow robots.txt, or at least certainly no technical requirement. I don't think there's any automatic legal requirement either. Maybe sites could add "you must honor policies set in robots.txt" to something like a terms of service but I have no idea if that would have enough teeth for a crawler to give up.

Cloudflare snd their customera have been desperately for years trying to kill scrapers in court. This is all. Meaningless, but they are probably gearing up for another legal battle to define robots.txt as a legal contract. Theyre going to use this marketplace theyre scamming people with to do it. They will fail.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#72

Post by haiku2077 »

deepsiml wrote: post #39
Meekro wrote: post #32

I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…

Not much into that kind of DevOps. What is a good basic caching in this instance?

It comes down to:

1. Use the Cache-Control header to express how to cache your site correctly (https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac...)

2. Use a CDN service, or at least a caching reverse proxy, to serve most of the cacheable requests to reduce load on the (typically much more expensive) origin servers

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#73

Post by hombre_fatal »

account42 wrote: post #56
hombre_fatal wrote: post #26

Earlier quoted context omitted.

Next line: > The rule has also been expanded to include more signatures of AI bots that do not follow the rules. The Block AI Bots rule on the Super Bot Fight Mode page does filter out most bot traffic. I was getting 10x the traffic from bots than I was from users. It definitely doesn't rely on robots.txt or user agent. I had to write a page rule bypass just to let my own tooling work on my website after enabling it.

How many of those "bots" you are filtering are actually bots and how many are regular users buttflare has misidentified as bots?

Pretty simple to see this if you've run a website: compare your analytics pre-bot to post-bot to post-bot-blocker.

There is a clear moment where you land on AI bot radar. For my large forum, it was a month ago.

Overnight, "72 users are viewing General Discussion" turned into "1720 users".

40% requests being cached turned into 3% of requests are cached.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#74

Post by giancarlostoro »

bee_rider wrote: post #62
jasonthorsness wrote: post #14

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…

I wonder… Google scrapes for indexing and for AI, right? I wonder if they will eventually say: ok, you can have me or not, if you don’t want to help train my AI you won’t get my searches either. That’s a tough deal but it is sort of self-consistent.

"Embrace, Extend, Extinguish" Google's mantra. And yes, I know about Microsoft's history with that phrase ;) But Google has done this with email, browsers (Google has web apps that run fine on Firefox but request you use Chrome), Linux (Android), and I'm sure there's others I am forgetting about.

So yeah, I too could see them doing this.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#75

Post by kentonv »

chasd00 wrote: post #29
cratermoon wrote: post #16

I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. Has Cloudflare said anything about identifying the bad actors?

i've mentioned this in a couple replies so maybe i'm wrong but it's up to the client to obey robots.txt. Why would they not just ignore it? Unless there's some legal consequence not complying with robots.txt then why even follow it? There's no technical enforcement of the policies in the file, it's up to the client to honor them.

> There's no technical enforcement of the policies in the file, it's up to the client to honor them.

That's incorrect. Cloudflare does in fact enforce this at a technical level. Cloudflare has been doing bot detection for years and can pretty reliably detect when bots are not following robots.txt and then block them.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#76

Post by zargath »

Sounds very basic, sadly.

Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform.

I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for access, ask for content, pay for content, etc. I want to offer solutions for bots to consume my content, instead of having to choose between full or no access.

I am all for AI, but please try to do better. Right now the internet is about to be eaten up by stupid bot farms and served into chat screens. They dont want to refer back to their source and when they do its with insane error rates.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#77

Post by mattcollins »

ijk wrote: post #70
fennecfoxy wrote: post #31

Earlier quoted context omitted.

With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Goo…

What I want to know is if the flood of scraping everyone has been complaining about is coming from people trying to scrape for training or bots doing RAG search. I get that everyone wants data, but presumably the big players already scraped the web. Do they really need to do it again? Or is it bit players reproducing data that's likely already in the training set? Or is it really that valuable to have your own scrape…

I wondered about this, too.

Cloudflare have some recent data about traffic from bots (https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-cr...) which indicates that, for the time being, the overwhelming majority of the bot requests are for AI training and not for RAG.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#78

Post by TechDebtDevin »

zargath wrote: post #76

Sounds very basic, sadly. Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform. I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for acce…

This comment seems like it comes from a Cloudflare employee.

This is clearly the first step in cf building out a marketplace where they will (fail) at attempting to be the middleman in a useless market between crawlers and publishers.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#79

Post by fluidcruft »

postalcoder wrote: post #11

> When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. We already know companies like Perplexity are masking their traffic. I'm sure there's more than meets…

Cloudflare already knows how to make the web hell for people they don't like.

I read the robots.txt entries as those AI bots that will be not marked as "malicious" and that will have the opportunity to be allowed by websites. The rest will be given the Cloudflare special.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#80

Post by zargath »

TechDebtDevin wrote: post #78
zargath wrote: post #76

Sounds very basic, sadly. Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform. I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for acce…

This comment seems like it comes from a Cloudflare employee. This is clearly the first step in cf building out a marketplace where they will (fail) at attempting to be the middleman in a useless market between crawlers and publishers.

nah, disappointed cf customer
Post reply on HN