The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

Post by account42 » Mon, Sep 01, 2025, 9:26 AM UTC

lucumo wrote: post #379

RobSm wrote: post #363

Earlier quoted context omitted.

You can always stop bots. Add login/password. But people want their content to be accessible to as large audience as possible, but at the same time they don't want that data to be accessible to the same audience via other channels. logic. Bots are not consuming your data - humans are. At the end of the day humans will eventually read it and take actions. For example chatgpt will mention your site, the user will visit…

> And no, nothing was different before 2022. Just look at google, the largest bot scraping network in the world. Since 1996. I'm sorry, but this statement shows you have no recent experience with these crawlernets. Google, from the beginning, has done their best to work with server owners. They respect robots.txt. I think they were the first to implement Crawl-Delay. They crawl based on how often things change anyway…

Post by account42 » Mon, Sep 01, 2025, 9:36 AM UTC

BobaFloutist wrote: post #239

wilson090 wrote: post #176

Earlier quoted context omitted.

How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.

> information is free and available for anyone. Bots aren't people. You can want public water fountains without wanting a company attaching a hose to the base to siphon municipal water for corporate use, rendering them unusable for everyone else. You can want free libraries without companies using their employees' library cards to systematically check out all the books at all times so they don't need to wait if they…

Post by account42 » Mon, Sep 01, 2025, 10:10 AM UTC

notatoad wrote: post #82

gausswho wrote: post #71

Earlier quoted context omitted.

What we need is some legal teeth behind robots.txt. It won't stop everyone, but Big Corp would be a tasty target for lawsuits.

It wouldn’t stop anyone. The bots you want to block already operate out of places where those laws wouldn’t be enforced.

Post by account42 » Mon, Sep 01, 2025, 10:18 AM UTC

jeroenhd wrote: post #378

rsync wrote: post #182

Earlier quoted context omitted.

“But the reality is how can someone small protect their blog or content from AI training bots?” Why would you need to? If your inability to assemble basic HTML forces you to adopt enormous, bloated frameworks that require two full cores of a cpu to render your post… … or if you think your online missives are a step in the road to content creator riches … … then I suppose I see the problem. Otherwise there’s no proble…

I have had to disable at least one service because AI bots kept hitting it and it started impacting other stuff I was running that I am more interested in. Part of it was the CPU load on the database rendering dozens of 404s per second (which still required a database call), part of it was that the thumbnail images were being queried over and over again with seemingly different parameters for no reason. I'm sure ther…

Post by account42 » Mon, Sep 01, 2025, 10:46 AM UTC

h4ck_th3_pl4n3t wrote: post #471

TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

The problem's cause really is about _who_ has to pay for the traffic, and currently that's the hosting end. If you turn that model around, suddenly AI web scrapers have to behave and all the issues that we currently have are kind of solved(?), because there is no incentive to scrape datasets anymore that were put together by others, and there automatically will be a payment incentive instead to buy high-quality datas…