or we could just require postage. (HTTP status code 402)
one potential solution: https://www.l402.org/
The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
201–210 of 520 posts
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#202Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
What we need is some legal teeth behind robots.txt. It won't stop everyone, but Big Corp would be a tasty target for lawsuits.
The funny thing about the good old WWW is the first two W's stand for world-wide.
So
Which legal teeth?
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#203Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.
[dead]
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#204Earlier quoted context omitted.
How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.
The problem is not AI bot scraping, per se, but "AI bot scraping while disregarding all licenses and ethical considerations". Freedom, the word, while implies no boundaries, is always bound by ethics, mutual respect and "do no harm" principle. The moment you trip either one of these wires and break them, the mechanisms to counter it becomes active. Then we cry "but, freedom?!". Freedom also contains the consequences…
The problem isn't "AI bot scraping while disregarding all licenses and ethical considerations". The problem is "AI bot scraping while ignoring every good practice to reduce bandwidth usage".
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#205I would love to get off Cloudflare but there are no real good alternatives
AWS is an alternative no?
We were supposed to pentest a website on AWS WAF last week. We encountered three types of blocks:
1) hard block without having done any requests yet. No clue why. Same browser (Burp's built-in Chromium), same clean state, same IP address, but one person got a captcha and the other one didn't. It would just say "reload the page to try again" forever. This person simply couldn't use the site at all; not sure if that would happen if you're on any other browser, but since it allowed the other Burp Suite browser, that doesn't seem to be the trigger for this perma-ban. (The workaround was to clone the cookie state from the other consultant, but normal users won't have that option.)
2) captcha. I got so many captchas, like every 4th request. It broke the website (async functionality) constantly. At some point I wanted to try a number of passwords for an admin username that we had found and, to my surprise, it allowed hundreds of requests without captcha. It blocks humans more than this automated bot...
3) "this website is under construction" would sometimes appear. Similar to situation#1, but it seemed to be for specific requests rather than specific persons. Inputting the value "1e9" was fine, "1e999" also fine, but "1e99" got blocked, but only on one specific page (entering it on a different page was fine). Weird stuff. If it doesn't like whatever text you wrote on a support form, I guess you're just out of luck. There's no captcha or anything you can do about it (since it's pretending the website isn't online at all). Not sure if this was AWS or the customer's own wonky mod_security variant
I dread to think if I were a customer of this place and I urgently needed them (it's not a regular webshop but something you might need in a pinch) and the only thing it ever gives me is "please reload the page to try again". Try what again?? Give me a human to talk to, any number to dial!
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#206Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
You could run https://zadzmo.org/code/nepenthes/ to punish the AI scrapers.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#207Earlier quoted context omitted.
> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.
they're basically describing the tragedy of the commons, but if a handful of the people have bulldozers to rip up all the grass and trees. We can't have nice things because the powerful cannot be held accountable. The powerful are powerful due to their legal teams and money, and power is the ability to carve exceptions to rules.
Bingo. Thanks for clarifying exactly my point
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#208Earlier quoted context omitted.
How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.
How is it available for everyone if the AI bots bring down your server?
Rate-limits? Use a CDN? Lots of traffic can be a problem whether it's bots or humans.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#209Earlier quoted context omitted.
The website owner has rights too. Are you arguing they cannot choose to implement such gatekeeping to keep their site operating in a financially viable manner?
The first article of our constitution says people shall be treated equally in equal situations. I presume that most countries have similar clauses but, beyond legalese, it's also simply in line with my ethics to treat everyone equally There are people behind those connection requests. I don't try to guess on my server who is a bot and who is not; I'll make mistakes and probably bias against people who use uncommon se…
If by "our constitution" you mean the U.S. Constitution then no, it says nothing of the sort. The first article of the U.S. Constitution concerns the organization of the legislative branch. You may be referencing the Equal Protection and Due Process clauses, in the Fifth and Fourteenth amendments, but neither of those applies in this situation either since there are no laws or governmental actions at issue here, and random sites on the internet are not universally considered to be public accommodations. Even in the ADA context, the law isn't actually clear, since websites aren't specified anywhere in the text at the federal level and there's no SCOTUS precedent on point.
Some states are more stringent with their own disability regulations or state constitutions, but no state anywhere in the U.S. has a law that says every visitor to a website has to be treated equally.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#210I have zero issue with Ai Agents, if there's a real user behind there somewhere. I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI - it's really annoying realising that we're tying up several cpu cores on AI crawling. Less than on real users and google et al.
Same with me. If there is a real user behind the use of the AI agents and they do not make excessive accesses in order to do what they are trying to do, then I do not have a complaint (the use of AI agents is not something I intend, but that is up to whoever is using them and not up to me). I do not like the excessive crawling.
However, what is more important to me than AI agents, is that someone might want to download single files with curl, or use browsers such as Lynx, etc, and this should work.