The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

Post by csomar » Sat, Aug 30, 2025, 11:34 AM UTC

BoredPositron wrote: post #216

TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

We have thousands of engineers of these companies right here on hackernews and they cry and scream about privacy and data governance on every topic but their own work. If you guys need a mirror to do some self reflection I am offering to buy.

Post by crote » Sat, Aug 30, 2025, 12:11 PM UTC

msgodel wrote: post #405

BrenBarn wrote: post #362

Earlier quoted context omitted.

The thing is that rhinoceroses aren't well-behaved. Even if some small fraction of them in theory might be well-behaved, the effort of trying to account for that is too small to bother. If 99% of rhinoceroses aren't well-behaved, the simple and correct response is to ban them all, and then maybe the nice ones can ask for a special permit. You switch from allow-by-default to block-by-default. Similarly it doesn't make…

ISPs are supposed to disconnect abusive customers. The correct thing to do is probably contact the ISP. Don't complain about scraping, complain about the DDOS (which is the actual problem and I'm increasingly beginning to believe the intent.)

Post by crote » Sat, Aug 30, 2025, 12:23 PM UTC

akoboldfrying wrote: post #406

BrenBarn wrote: post #247

Earlier quoted context omitted.

No, that is not true. It is only true if you just equate "AI training bots" with "people" on some kind of nominal basis without considering how they operate in practice. It is like saying "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" Well, the reason is because rhinoceroses are simply not going to stroll up and down the aisles and head to the checkout line quietly wit…

> "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" What this scenario actually reveals is that the words "open to the public" are not intended to mean "access is completely unrestricted". It's fine to not want to give completely unrestricted access to something. What's not fine, or at least what complicates things unnecessarily, is using words like "open and free" to des…

Post by lostmsu » Sat, Aug 30, 2025, 12:41 PM UTC

spankibalt wrote: post #337

wvenable wrote: post #250

Earlier quoted context omitted.

You have a problem with badly behaved scrapers, not AI. I can't disagree with being against badly behaved scrapers. But this is neither a new problem or an interesting one from the idea of making information freely available to everyone, even rhinoceroses, assuming they are well behaved. Blocking bad actors is not the same thing as blocking AI.

> You have a problem with badly behaved scrapers, not AI. And you have a problem understanding that "freedom and openness" extend only to where the rights (e. g. the freedom) of another legal entity begins. When I don't want "AI" (not just the badly-behaved subset) rifling my website then I should be well within my rights to disallow just that, in the same way as it's your right to allow them access to your playgroun…

Post by lostmsu » Sat, Aug 30, 2025, 12:53 PM UTC

Symbiote wrote: post #396

immibis wrote: post #340

Earlier quoted context omitted.

There's actually not much evidence of this, since the attack traffic is anonymous.

HN people working in these AI companies have commented to say they do this, and the timing correlates with the rise of AI companies/funding. I haven't tried to find it in my own logs, but others have said blocking an identifiable AI bot soon led to the same pattern of requests continuing through a botnet.