Live data from Hacker News

AI bots are destroying Open Access

go-to-hellman.blogspot.com

31–40 of 52 posts

Re: AI bots are destroying Open Access

#31
The way I see it, there may be pain now, but this is just the chaos before the web eventually becomes the semantic web, as mostly conceived by Berners-Lee[0]. Make all data available for true, easy, permanent open access instead of throwing up roadblocks to be circumvented, so all organizations wanting to train AI models can access, and thus properly democratize the ecosystem. It's that or end up with a handful of players with pockets deep enough to do what it takes to get the data, and then gatekeep the results for their own profit.

Also, eventually I see most people filtering their queries through something like Perplexity anyway instead of going to individual sites, so those putting up barriers will lose out on human traffic in any case. Let's ensure that the results people are able to access via AI continually improves so the "slop" term disappears that much faster.

[0] https://en.m.wikipedia.org/wiki/Semantic_Web

Re: AI bots are destroying Open Access

#32
post #5

Earlier quoted context omitted.

AI bots operators are financially incentivized to not be good citizens, they want as much data as possible as fast as possible and don't care who they piss off in the process. Plus for now at least they have effectively unlimited money to throw at bandwidth, storage, IP addresses, crawling with full-blown headless browsers, etc.

And it gets worse. For now they are probably paying to use residential IP addresses that they get from other services that sell them (and these services get them from people who willingly sell some of their bandwidth for cents). But I think it won't be long before we start seeing the AI companies having each their own swarm of residential IP addresses by selling themselves a browser extension or mobile app, saying so…

I thought this as well reading the last discussion. I believe some extra shady free VPNs have used a browser extension to borrow your endpoint to work around geoblocks, etc. I always thought this was a terrible idea, who wants their home internet ip associated with some random VPN users traffic? A voracious mindless bot that slurps up everything it can get to isn't much better.

Microsoft could build this into Windows even, they already use your upload bandwidth to help distribute their updates.

Re: AI bots are destroying Open Access

#33

The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal…

Anubis (or something similar) is an alternative option: https://github.com/TecharoHQ/anubis

Aside from the obvious disadvantages of a non-anoynmous web I also don't even think it will work. How do you deal with identification and punishment of threat actors across the globe? We've been failing at that since the start. When was the internet ever high trust?

Re: AI bots are destroying Open Access

#34

The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal…

Anubis (or something similar) is an alternative option: https://github.com/TecharoHQ/anubis Aside from the obvious disadvantages of a non-anoynmous web I also don't even think it will work. How do you deal with identification and punishment of threat actors across the globe? We've been failing at that since the start. When was the internet ever high trust?

>When was the internet ever high trust?

In the 1970s and 1980s.

Re: AI bots are destroying Open Access

#35
post #14

The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal…

I don't know. Once I know the who the legal entity is who I assert is a bad actor, I'm not sure there is really an recourse to be had. Your honor, these people are visiting my website in a way that makes me sad? I feel that we would need to encode bad behavior in a legally reasonable way first. And not to mention that you'll have to bring legal disputes a legal entity at a time. And some of these legal entities have…

[deleted]

Re: AI bots are destroying Open Access

#36

I have temporarily disabled my HTTP server for now. (I set up port knocking for a day, but I got rid of it due to a kernel panic.) My issue is not to prevent anyone from obtaining a copy if they want to do, and I want to ensure that users can use curl, Lynx, and other programs; I do not want to require JavaScripts, CSS, Firefox, Google, etc. My problem is that these LLM scraping bots are badly behaved, making many re…

How can you tell they are LLM bots?

[deleted]

Re: AI bots are destroying Open Access

#37
post #4

Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?

Although I think it's likely that these are "AI" bots, the real problem is the proliferation of rich and crappy crawlers. Whether or not legacy crawlers respect robots.txt, etc., they do seem to be sophisticated enough to determine when they're stuck in a loop. The home organizations of these new crawlers seem to have more money than sense and are often getting stuck in large dynamic sites for months without retrieving any new information. Among all of the articles about building "bot traps," libraries realized that they have unwittingly been in the bot trapping business for years.

Re: AI bots are destroying Open Access

#38
post #4

Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?

I personally don't care the intended use of the crawling -- and also don't know that the bots we are seeing now are "AI bots", I would not have used that phrase.

What many of us have seen is a huge increase in bot crawling traffic, from highly distributed IPs, and often requesting insane combinations of query params that don't actually get them useful content -- that bring down our sites. (And that increase their volume if you scale up your resources!) They seem to have very deep pockets, in that they don't mind that they are scraping terrabytes of useless/duplicate content from me (they can get all the actual useful open content from my SiteMap and I wouldn't mind!)

That's what bothers me. I don't care if they scrape my site for AI purposes in polite robots.txt-respecting honest-user-agent low-volume ways. And if they are doing it the way they are doing it for something other than AI, it's just as much of a problem. (The best guess is just that it's for AI).

So I agree with you that I wouldn't have spoken of this in terms of "AI".

But it has become a huge problem.

"Fighting the AI scraperbot scourge" https://lwn.net/Articles/1008897/

"LLM crawlers continue to DDoS SourceHut" https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/

"Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries" https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-domi...

Some of us -- we think jokingly -- wonder if Cloudflare or other WAF purveyors are behind it. It is leaving most of us no choice but some kind of WAF or bot detection.

Re: AI bots are destroying Open Access

#39

I guess the problem of throttling connections to human rates is that the bots rapidly eat up all of the connections. Can we at least get rid of CAPTCHAs now since they clearly don't work?

I don't mean to be marketting for them, but the CloudFlare Turnstile "captcha alternative" (Similar to Google ReCaptcha and others) has been working for me. it's the only thing that has of what I tried so far (rate-limiting IPs, fail2ban, etc -- their IPs are just too distributed).

It doesn't make the user do a puzzle, it's the kind that either works entirely automatically or in some cases asks the user to tick a checkbox. You have probably seen it proliferating across the internet in your personal use becuase, well, see above.

Re: AI bots are destroying Open Access

#40
post #19

Earlier quoted context omitted.

As much as I like BitTorrent, people (usually) don't want to provide open access to information; what they (usually) want is to be an "open" gateway to that information, as long as they are the centralized point of distribution whose name appears in the URL bar, and as long as they control when they can remove access to that information. Creating a torrent is not showy enough, because the credit is "just" another fil…

It doesn't matter what people usually want. It's sufficient for someone to want to torrent the open-access articles, even if everyone else is playing the exploitative games you're describing. The Berlin Declaration that defined "open access" https://openaccess.mpg.de/Berlin-Declaration requires specifically > The author(s) and right holder(s) of such contributions grant(s) to all users a free, irrevocable, worldwide,…

Someone still needs to assemble those documents to create the torrent collections in the first place. That's harder now that captchas and other access walls are getting more and more hostile to human consumption.

So, yes, torrents help to preserve what has already been archived in the past, but we still need a lot more works to be deposited in open archives like Zenodo or arxiv in the first place.

Post reply on HN