Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

positiveblue.substack.com

Post reply on HN
Print view View original

361–370 of 520 posts

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#361

Post by DanOpcode »

eldenring wrote: post #57
TIPSIO wrote: post #20

Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…

I recently found out my website has been blocked by AI agents, when I had never asked for it. It seems to be opt-out by default, but in an obscure way. Very frustrating. I think some of these companies (one in particular) are risking burning a lot of goodwill, although I think they have been on that path for a while now.

Are you talking about Cloudflare? The default seems indeed to be to block AI crawlers when you set up a new site with them.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#362

Post by BrenBarn »

wvenable wrote: post #250
BrenBarn wrote: post #247

Earlier quoted context omitted.

No, that is not true. It is only true if you just equate "AI training bots" with "people" on some kind of nominal basis without considering how they operate in practice. It is like saying "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" Well, the reason is because rhinoceroses are simply not going to stroll up and down the aisles and head to the checkout line quietly wit…

You have a problem with badly behaved scrapers, not AI. I can't disagree with being against badly behaved scrapers. But this is neither a new problem or an interesting one from the idea of making information freely available to everyone, even rhinoceroses, assuming they are well behaved. Blocking bad actors is not the same thing as blocking AI.

The thing is that rhinoceroses aren't well-behaved. Even if some small fraction of them in theory might be well-behaved, the effort of trying to account for that is too small to bother. If 99% of rhinoceroses aren't well-behaved, the simple and correct response is to ban them all, and then maybe the nice ones can ask for a special permit. You switch from allow-by-default to block-by-default.

Similarly it doesn't make sense to talk about what happens if AI bots were well-behaved. If they are, then maybe that would be okay, but they aren't, so we're not talking about some theoretical (or past) situation where bots were well-behaved and scraped in a non-disruptive fashion. We're talking about the present reality in which there actually are enormous numbers of badly-behaved bots.

Incidentally, I see that in a lot of your responses on this thread you keep suggesting that people's problem is "not with AI" but with something else. But look at your comment that I initially replied to:

> Blocking AI training bots is not free and open for all.

We're not talking about "AI". We're talking about AI training bots. If people want to develop AI as a theoretical construct and train it on datasets they download separately in a non-disruptive way, great. (Well, actually it's still terrible, but for other reason. :-) ) But that's not what people are responding to in this thread. They're talking about AI training bots that scrape websites in a way that is objectively more harmful than previous generations of scrapers.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#363

Post by RobSm »

BrenBarn wrote: post #247
wvenable wrote: post #172

Earlier quoted context omitted.

> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.

No, that is not true. It is only true if you just equate "AI training bots" with "people" on some kind of nominal basis without considering how they operate in practice. It is like saying "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" Well, the reason is because rhinoceroses are simply not going to stroll up and down the aisles and head to the checkout line quietly wit…

You can always stop bots. Add login/password. But people want their content to be accessible to as large audience as possible, but at the same time they don't want that data to be accessible to the same audience via other channels. logic. Bots are not consuming your data - humans are. At the end of the day humans will eventually read it and take actions. For example chatgpt will mention your site, the user will visit it.

And no, nothing was different before 2022. Just look at google, the largest bot scraping network in the world. Since 1996.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#364

Post by frm88 »

wvenable wrote: post #298
danudey wrote: post #289

Earlier quoted context omitted.

Yes, because a top hacker news post takes your server down because a large number of actual humans are looking to gain actual value from your posts. Meanwhile, you stand to benefit from the HN discussion by learning new things and perspectives from the community. The AI bot assault, on the other hand, is one company (or a few companies) re-fetching the same data over and over again, constantly, in perpetuity, just in…

But then we get to use those AI tools. The refrain here comes down not to "AI" but mostly to "the AI bot assault" which is a different thing. Sure lets have an discussion about badly behaved and overzealous web scrapers. As for credit, I've asked AI for it's references and gotten them. If my information is merely mushed into AI training model I'm not sure why I need credit. If you discuss this thread with your friend…

"If you discuss this thread with your friends are you going to give me credit?"

Yes. How else would I enable my friends to look it up for themselves?

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#365

Post by somerandom2407 »

rustc wrote: post #109
SoftTalker wrote: post #102

Earlier quoted context omitted.

If you put your information freely on the web, you should have minimal expectations on who uses it and how. If you want to make money from it, put up a paywall. If you want the best of both worlds, i.e. just post freely but make money from ads, or inserting hidden pixels to update some profile about me, well good luck. I'll choose whether I want to look at ads, or load tracking pixels, and my answer is no.

> If you put your information freely on the web, you should have minimal expectations on who uses it and how. Does this only apply to "information" or should we treat all open source code as public domain?

In a lot of circumstances, that is exactly the case. What the open source license stops is redistribution under terms that violate the license, not usage itself. An individual can very well take your open source code, make any changes they want, compile and use it for their own purposes without adhering to the terms of your license - as long as they don't redistribute it.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#366

Post by somerandom2407 »

verdverm wrote: post #136
SoftTalker wrote: post #102

Earlier quoted context omitted.

If you put your information freely on the web, you should have minimal expectations on who uses it and how. If you want to make money from it, put up a paywall. If you want the best of both worlds, i.e. just post freely but make money from ads, or inserting hidden pixels to update some profile about me, well good luck. I'll choose whether I want to look at ads, or load tracking pixels, and my answer is no.

I'm not talking about ads or pixels, I'm referring to bot operators creating so much traffic that the network bill makes the hosting financially impossible > my answer is no. Rights for me, but not for thee?

You have every right to take the content offline, or to put any technical barriers you desire in place to access it - but that's about all you should be able to do.

If you don't want to lose money and don't feel confident that you can protect your content with technical measures, best to take your stuff off the internet.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#367

Post by [deleted] »

theamk wrote: post #190
supriyo-biswas wrote: post #132

Earlier quoted context omitted.

The actual response to which Anubis was created is seemingly a strange kind of DDOS attack that has been misattributed to LLMs, but is some kind of attacker that makes partial GET requests that are aborted soon after sending the request headers, mostly coming from residential proxies. (Yes, it doesn’t help that the author of Anubis also isn’t fully aware of the mechanics of the attack. In fact, there is no proper wri…

> a strange kind of DDOS attack that has been misattributed to LLMs, , but is some kind of attacker that makes partial GET requests that are aborted soon after sending the request headers, mostly coming from residential proxies. um, no? Where did you get this strange bit of info. The original reports say nothing of that sort: https://news.ycombinator.com/item?id=42790252 ; and even original motivation for Anubis was…

[deleted]

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#368

Post by tsimionescu »

wvenable wrote: post #324
bayindirh wrote: post #308

Earlier quoted context omitted.

Let me try to make my point as compact as possible. I may fail, but please bear with me. I prefer Free Software to Open Source software. My license of choice is A/GPLv3+. Because, I don't want my work to be used by people/entities in a single sided way. The software I put out is the software I develop for myself, with the hope of being useful for somebody else. My digital garden is the same. My blog is a personal dia…

I respect the desire for reciprocity, but strong copyleft isn't the only, or even the best, way to protect user freedom or public knowledge. My opinion is that permissive licensing and open access to learn from public materials have created enormous value precisely because they don't pre-empt future uses. Requiring permission for every new kind of reuse (including ML training) shrinks the commons, entrenches incumben…

Your whole argument is easily addressed by requiring the AI models to be open source. That way, they obviously respect the AGPL and any other open license, and contribute to the information being kept free. Letting these companies knowingly and obviously infringe licenses and all copyright as they do today is obviously immoral, and illegal.

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#369

Post by tsimionescu »

wvenable wrote: post #298
danudey wrote: post #289

Earlier quoted context omitted.

Yes, because a top hacker news post takes your server down because a large number of actual humans are looking to gain actual value from your posts. Meanwhile, you stand to benefit from the HN discussion by learning new things and perspectives from the community. The AI bot assault, on the other hand, is one company (or a few companies) re-fetching the same data over and over again, constantly, in perpetuity, just in…

But then we get to use those AI tools. The refrain here comes down not to "AI" but mostly to "the AI bot assault" which is a different thing. Sure lets have an discussion about badly behaved and overzealous web scrapers. As for credit, I've asked AI for it's references and gotten them. If my information is merely mushed into AI training model I'm not sure why I need credit. If you discuss this thread with your friend…

No, you don't "get to" use the AI tools. You have to buy access to them (beyond some free trials).

Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch

#370

Post by 1gn15 »

krapp wrote: post #352
1gn15 wrote: post #347

Earlier quoted context omitted.

Yes, it is an unreasonable and absurd ask. You cannot want freedom while restricting it. You forget that it is people that use AI agents, essentially, being cyborgs. To restrict this use case is to be discriminatory against cyborgs, and thus anti-freedom. We are lucky that there is no way to detect it.

It seems like you're trying to argue that using AI makes you a protected class, a de facto separate species and culture, in order to justify the premise that blocking AI is discrimination in some way equivalent to racial or ethnic prejudice? If so, no. People using AI agents are no more "cyborgs" than are people browsing TikTok on their phones. You're just a regular human using software, the software is not you and d…

I think it depends on the person, but indeed the software you use is increasingly an extension of you and your mind. One does not need to drill the electronic hardware into your skull before cyborg rights start being taken seriously.

Also, I'm not a human.

Post reply on HN