Are bots using a large number of IP addresses simultaneously, so they look like a DDOS attack? Or are they just making ordinary requests from a small number of addresses. If it's the latter, all you need is some kind of fair queuing so those requests compete with each other for access, not with other users.
The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
281–290 of 520 posts
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#282The web doesn't need attestation. It doesn't need signed agents. It doesn't need Cloudflare deciding who's a "real" user agent. It needs people to remember that "public" means PUBLIC and implement basic damn rate limiting if they can't handle the traffic. The web doesn't need to know if you're a human, a bot, or a dog. It just needs to serve bytes to whoever asks, within reasonable resource constraints. That's it. Th…
[deleted]
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#283Earlier quoted context omitted.
What we need is stop fighting robots and start welcoming and helping them. I se zero reasons to oppose robots visiting any website I would build. The only purpose I ever tried disallowed robots for was preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. Now I think we should write separate instructions for different kinds of robots: a search engine…
> I see zero reasons to oppose robots visiting any website I would build. > preventing search engines from indexing incomplete versions or going the paths which really make no sense for them to go. What will you do when the bots ignore your instructions, and send a million requests a day to these URLs from half a million different IP addresses?
Sue them / press charges. DDoS is a felony.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#284Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
The dream is real, man. If you want open content on the Internet, it's never been a better time. My blog is open to all - machine or man. And it's hosted on my home server next to me. I don't see why anyone would bother trying to distinguish humans from AI. A human hitting your website too much is no different from an AI hitting your website too much. I have a robots.txt that tries to help bots not get stuck in loops…
> I don't see why anyone would bother trying to distinguish humans from AI.
Because a hundred thousand people reading a blog post is more beneficial to the world than an AI scraper bot fetching my (unchanged) blog post a hundred thousand times just in case it's changed in the last hour.
If AI bots were well-behaved, maintained a consistent user agent, used consistent IP subnets, and respected robots.txt, I wouldn't have a problem with them. You could manage your content filtering however you want (or not at all) and that would be that. Unfortunately at the moment, AI bots do everything they can to bypass any restrictions or blocks or rate limits you put on them; they behave as though they're completely entitled to overload your servers in their quest to train their AI bots so they can make billions of dollars on the new AI craze while giving nothing back to the people whose content they're misappropriating.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#285Earlier quoted context omitted.
What we need is some legal teeth behind robots.txt. It won't stop everyone, but Big Corp would be a tasty target for lawsuits.
It should have the same protections as an EULA, where the crawler is the end user, and crawlers should be required to read it and apply it.
So none at all? EULAs are mostly just meant to intimidate you so you won't exercise your inalienable rights.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#286Earlier quoted context omitted.
No, that is not true. It is only true if you just equate "AI training bots" with "people" on some kind of nominal basis without considering how they operate in practice. It is like saying "If your grocery store is open to the public, why is it not open to this herd of rhinoceroses?" Well, the reason is because rhinoceroses are simply not going to stroll up and down the aisles and head to the checkout line quietly wit…
You have a problem with badly behaved scrapers, not AI. I can't disagree with being against badly behaved scrapers. But this is neither a new problem or an interesting one from the idea of making information freely available to everyone, even rhinoceroses, assuming they are well behaved. Blocking bad actors is not the same thing as blocking AI.
Badly behaved scrapers are not a new problem, but badly behaved scrapers run by multibillion-dollar companies which use every possible trick to bypass every block or restriction or rate limit you put in front of them is a completely new problem on a scale we've never seen before.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#287Earlier quoted context omitted.
Why was it OK for Google to incorporate their words into a for-profit search index which has increasingly sucked all the profit out of the system? My Ithaca friends on Facebook complain incessantly about the very existence of AI to the extent that I would not want to say I ask Copilot how to use Windows Narrator or Junie where the CSS that makes this text bold or sometimes have Photoshop draw an extra row of bricks i…
They were okay with it when Google was sending them traffic. Now they often don’t. They’ve broken the social contract of the web. So why should the sites whose work is being scraped be expected to continue upholding their end?
Not only are they scraping without sending traffic, they're doing so much more aggressively than Google ever did; Google, at least, respected robots.txt and kept to the same user-agent. They didn't want to index something that a server didn't want indexed. AI bots, on the other hand, want to index every possible thing regardless of what anyone else says.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#288Earlier quoted context omitted.
> You make unauthenticated requests cheap enough that you don't care about volume. In the days before mandatory TLS it was so easy to set up a Squid proxy on the edge of my network and cache every plain-HTTP resource for as long as I want. Like yeah, yeah, sure, it sucked that ISPs could inject trackers and stuff into page contents, but I'm starting to think the downsides of mandatory TLS outweigh the upsides. We mad…
You can still do that, you have the caching reverse proxy at the edge of the network be the thing that terminates TLS.
Not really. At minimum you will break all of these sites on the HSTS preload list: https://source.chromium.org/chromium/chromium/src/+/main:net...
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#289Earlier quoted context omitted.
How is it available for everyone if the AI bots bring down your server?
Is that really the problem we are discussing? I've had people attack my server and bring it down. But that has nothing to do with being free and open to everyone. A top hacker news post could take my server.
Yes, because a top hacker news post takes your server down because a large number of actual humans are looking to gain actual value from your posts. Meanwhile, you stand to benefit from the HN discussion by learning new things and perspectives from the community.
The AI bot assault, on the other hand, is one company (or a few companies) re-fetching the same data over and over again, constantly, in perpetuity, just in case it's changed, all so they can incorporate it into their training set and make money off of it while giving you zero credit and providing zero feedback.
Re: The web does not need gatekeepers: Cloudflare’s new “signed agents” pitch
#290Earlier quoted context omitted.
How is it available for everyone if the AI bots bring down your server?
Rate-limits? Use a CDN? Lots of traffic can be a problem whether it's bots or humans.
You realize this entire thread is about a pitch from a CDN company trying to solve an issue that has presented itself at such a scale that this is the best option they can think of to keep the web alive, right?
"Use a CDN" is not sufficient when these bots are so incredibly poorly behaved, because you're still paying for that CDN and this bad behavior is going to cost you a fortune in CDN costs (or cost the CDN a fortune instead, which is why Cloudflare is suggesting this).