Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI bots are destroying Open Access

go-to-hellman.blogspot.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: AI bots are destroying Open Access

#11

Post by flakeoil »

xnx wrote: post #4

Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?

They are maxing out the CPU of the web servers. For example Anthropic hitting a server 11 times per second non-stop easily loads a basic web server serving a dynamic website. That's like 1 million page views per day. And they continue for weeks even though they could have scraped whatever they are after in less than an hour.

Re: AI bots are destroying Open Access

#12

Post by JohnFen »

nathanaldensr wrote: post #2

The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal…

The truth and tragedy of this is very clear to me. I am hoping this is something that will eventually be solved, but I don't expect it. These companies are on a burn-and-pillage rampage.

Re: AI bots are destroying Open Access

#13

Post by zzo38computer »

I have temporarily disabled my HTTP server for now. (I set up port knocking for a day, but I got rid of it due to a kernel panic.)

My issue is not to prevent anyone from obtaining a copy if they want to do, and I want to ensure that users can use curl, Lynx, and other programs; I do not want to require JavaScripts, CSS, Firefox, Google, etc.

My problem is that these LLM scraping bots are badly behaved, making many requests and repeating them even though there is no good reason to do so, and potentially overloading the servers. These things are mentioned in the article. Some bots are not so badly behaved, and those are not the problem.

Re: AI bots are destroying Open Access

#14

Post by Verdex »

nathanaldensr wrote: post #2

The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal…

I don't know. Once I know the who the legal entity is who I assert is a bad actor, I'm not sure there is really an recourse to be had.

Your honor, these people are visiting my website in a way that makes me sad? I feel that we would need to encode bad behavior in a legally reasonable way first.

And not to mention that you'll have to bring legal disputes a legal entity at a time. And some of these legal entities have very deep pockets.

Unless the suggestion is that internet providers are all going to join together to stand up for the little guy? Somehow I'm not optimistic.

(Finally IPv6 has taken decades to get to where it is today. Somehow I don't see legally attributable IP traffic extension to be ready and deployed any faster)

Re: AI bots are destroying Open Access

#15

Post by rcxdude »

xnx wrote: post #4

Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?

Seems like the latter. There's basically a large number of well-funded attempts to crawl the internet, and enough of them are badly behaved enough it's basically a DDOS against smaller hosts.

Re: AI bots are destroying Open Access

#16

Post by quectophoton »

kragen wrote: post #9

Sheesh, just use BitTorrent. That's what open access licensing is for! BitTorrent's tit-for-tat approach limits the harm selfish actors can do, only greatly rewarding those whose behavior benefits others, and has been shown to be very robust against active disruption attempts for decades now. Moreover, it also confers some resistance to falsification of the published record, to linkrot, and to publishing companies go…

As much as I like BitTorrent, people (usually) don't want to provide open access to information; what they (usually) want is to be an "open" gateway to that information, as long as they are the centralized point of distribution whose name appears in the URL bar, and as long as they control when they can remove access to that information.

Creating a torrent is not showy enough, because the credit is "just" another file and/or a comment in the torrent metadata.

Granted, they usually do that because they want to "kindly" advertise a way to donate to them (EDIT: or to track you, or other similar goals), and there's nothing wrong with trying to get donations, but there's clearly a conflict of interest at play here.

Re: AI bots are destroying Open Access

#17

Post by paulddraper »

zzo38computer wrote: post #13

I have temporarily disabled my HTTP server for now. (I set up port knocking for a day, but I got rid of it due to a kernel panic.) My issue is not to prevent anyone from obtaining a copy if they want to do, and I want to ensure that users can use curl, Lynx, and other programs; I do not want to require JavaScripts, CSS, Firefox, Google, etc. My problem is that these LLM scraping bots are badly behaved, making many re…

How can you tell they are LLM bots?

Re: AI bots are destroying Open Access

#18

Post by quectophoton »

jsheard wrote: post #5
xnx wrote: post #4

Why are "AI" bots generating so much fuss. Is it because there are so many of them? Is it because AI companies are each writing their own (bad) crawlers instead of using existing ones?

AI bots operators are financially incentivized to not be good citizens, they want as much data as possible as fast as possible and don't care who they piss off in the process. Plus for now at least they have effectively unlimited money to throw at bandwidth, storage, IP addresses, crawling with full-blown headless browsers, etc.

And it gets worse.

For now they are probably paying to use residential IP addresses that they get from other services that sell them (and these services get them from people who willingly sell some of their bandwidth for cents).

But I think it won't be long before we start seeing the AI companies having each their own swarm of residential IP addresses by selling themselves a browser extension or mobile app, saying something like:

"Get faster results (or a discount) by using our extension! By using your own internet connection to fetch the required context, you won't need to share computing resources with other users, thusly increasing the speed of your queries! Plus, since you don't use our servers, that means we can pass our savings to you as a discount!"

Then in small letter saying they use your connection for helping others with their queries, or being more eco-friendly because sharing, or whatever they come up with to justify this.

Re: AI bots are destroying Open Access

#19

Post by kragen »

quectophoton wrote: post #16
kragen wrote: post #9

Sheesh, just use BitTorrent. That's what open access licensing is for! BitTorrent's tit-for-tat approach limits the harm selfish actors can do, only greatly rewarding those whose behavior benefits others, and has been shown to be very robust against active disruption attempts for decades now. Moreover, it also confers some resistance to falsification of the published record, to linkrot, and to publishing companies go…

As much as I like BitTorrent, people (usually) don't want to provide open access to information; what they (usually) want is to be an "open" gateway to that information, as long as they are the centralized point of distribution whose name appears in the URL bar, and as long as they control when they can remove access to that information. Creating a torrent is not showy enough, because the credit is "just" another fil…

It doesn't matter what people usually want. It's sufficient for someone to want to torrent the open-access articles, even if everyone else is playing the exploitative games you're describing. The Berlin Declaration that defined "open access" https://openaccess.mpg.de/Berlin-Declaration requires specifically

> The author(s) and right holder(s) of such contributions grant(s) to all users a free, irrevocable, worldwide, right of access to, and a license to copy, use, distribute, transmit and display the work publicly and to make and distribute derivative works, in any digital medium for any responsible purpose, subject to proper attribution of authorship (community standards, will continue to provide the mechanism for enforcement of proper attribution and responsible use of the published work, as they do now), as well as the right to make small numbers of printed copies for their personal use.

This guarantees that such torrents are legal unless the original authors are infringing copyright.

So there is no danger of AI bots destroying open access.

Post reply on HN