Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

251–260 of 279 posts

Re: We can't have nice things because of AI scrapers

#251

Post by jijijijij »

lxgr wrote: post #250
jijijijij wrote: post #239

Earlier quoted context omitted.

No, but subnets can't be as easily associated with unwanted traffic. If IPv6 gets blocked you just get another IP. A VPN or hosting provider can't simply rent, or god forbid buy IPv4 addresses and subnets, arbitrarily. The IPs they use are rather static and easy to discover. Rather trivial to block all them, preemptively. Residential IPv4 VPNs are not legal offerings and their use is limited. VPNs can fight traffic a…

> VPNs can fight traffic analysis, they can't fight preemptive IPv4 blocking. How do you think VPNs are getting past VOD providers’ VPN block lists? > Residential IPv4 VPNs are not legal offerings and their use is limited. What’s illegal about them? And does it matter to uncooperative/aggressive bots?

> How do you think VPNs are getting past VOD providers’ VPN block lists?

In my experience, they most often don't. If you got more insights, please enlighten me. I presume VPNs which get past VPN block lists, are just not yet on the radar, or don't provide the privacy claimed, not actually fully in control of their infrastructure.

> What’s illegal about them?

Where do you think residential IPs are coming from? It's often botnets or otherwise compromised devices, or people tricked into sharing their connection. In any case, it's most certainly breaking the ISP's TOS. Because of the effort behind providing residential IPs, these VPN services are rather expensive. And certainly not trustworthy in regard to privacy. If offering residential IPs would be legal, every VPN service would provide them.

> And does it matter to uncooperative/aggressive bots?

No. They are used for mostly shady/criminal activity, where the limitations and legality don't matter. I doubt commercial LLM crawlers and data intense campaigns aren't bothered by legality, stability, connectivity or (upload) bandwidth limitations. Like, you wouldn't crawl the web on a mobile connection.

Re: We can't have nice things because of AI scrapers

#252

Post by tclancy »

tommek4077 wrote: post #108
tclancy wrote: post #61

Earlier quoted context omitted.

Yes yes, definitely people don’t know what they’re doing and not that they’re operating on a scale or problem you are not. Metabrainz cannot cache all of these links as most of them are hardly ever hit. Try to assume good intent.

But serving HTML is unbelievably cheap, isn't it?

Run 72,000 database queries to generate a bunch of random HTML files no one has asked for in five years is not, especially compared to downloading the files designed for it.

Re: We can't have nice things because of AI scrapers

#253

Post by TeMPOraL »

philipwhiuk wrote: post #162
TeMPOraL wrote: post #131

Earlier quoted context omitted.

LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access . The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.

And rarely cite their sources, thus affording the author not so much a crumb of benefit in kind.

Which is irrelevant if you're truly trying to "pay it forward".

That is the core of my observation: people claim to publish to benefit society, but push come to shove, they care more about getting credit and having oversight over who is benefiting, to the point of refusing to publish further (and sometimes unpublishing things) if that credit/control isn't given.

The problem isn't in wanting these things - it's in not being up-front about it.

Re: We can't have nice things because of AI scrapers

#254

Post by TeMPOraL »

soco wrote: post #220
fartfeatures wrote: post #137

Earlier quoted context omitted.

I realise you are making assertions for which you have no evidence. Until a standard exists we can't just assume nobody will use it, particularly when it makes the very task they are scraping for simpler and more efficient.

Simpler and efficient for who? I imagine some random guy vibe coding "hi chatgpt I want to scrape this and this website", getting something running, then going to LinkedIn to brag about AI. Yes I have no hard evidence for this, but I see things on LinkedIn.

That's not the problem being discussed here, though. That's normal usage, and you can hardly blame AI companies for shitty scrapers random users create on demand, because it's merely a symptom of coding getting cheap. Or, more broadly, the flip side of the computer becoming an actual "bicycle for the mind" and empowering end-users for a change.

Re: We can't have nice things because of AI scrapers

#255

Post by johnnyanmac »

fartfeatures wrote: post #137
ethin wrote: post #132

Earlier quoted context omitted.

You do realize that these AI scrapers are most likely written by people who have no idea what they're doing right? Or they just don't care? If they were, pretty much none of the problems these things have caused would exist. Even if we did standardize such a thing, I doubt they would follow it. After all, they think they and everyone else has infinite resources so they can just hammer websites forever.

I realise you are making assertions for which you have no evidence. Until a standard exists we can't just assume nobody will use it, particularly when it makes the very task they are scraping for simpler and more efficient.

A lot of the internet is built on trust. Mix in this article describing yet another tragedy of the Commons and you can see where this logically ends up as.

Unless we have some government enforcing the standard, another trust based contract won't do much.

Re: We can't have nice things because of AI scrapers

#259

Post by TeMPOraL »

johnnyanmac wrote: post #255
fartfeatures wrote: post #137

Earlier quoted context omitted.

I realise you are making assertions for which you have no evidence. Until a standard exists we can't just assume nobody will use it, particularly when it makes the very task they are scraping for simpler and more efficient.

A lot of the internet is built on trust. Mix in this article describing yet another tragedy of the Commons and you can see where this logically ends up as. Unless we have some government enforcing the standard, another trust based contract won't do much.

> A lot of the internet is built on trust.

Yes. In this context, the problem is that you cannot trust websites to provide a standardized bulk download options. Most of them have (often pretty selfish or user-abusive) reasons not to provide any bulk download, much less proactively conform to some bottom-up standards. As a result, unless one is only targeting one or few very specific sites, even thinking about making the scrapper support anything but the standard crawling approach costs more in developer time than the benefit it brings.

Re: We can't have nice things because of AI scrapers

#260

Post by danaris »

expedition32 wrote: post #219
danaris wrote: post #210

Earlier quoted context omitted.

> That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor. Well, right; that's the problem. They take up orders of magnitude more resources. They absolutely hammer the server. They don't care if your website even survives , so long as they get every single drop of data they can for training. Source: my own personal experience with them taking down my tiny brow…

This is like email were eventually 90% of it was spam and we all got spam filters.

Except that something effectively equivalent to spam filters will be utterly ineffective here.

Spam filters

- mitigate the symptom (our inboxes being impossible to trawl through for real emails)

- reduce the incentive (because any spam mail that isn't seen by a human being reduces the chances they'll profit from their spamming)

- but does not affect the resource consumption directly (because the email has already been sent through the internet)

Now, this last point barely matters with spam, because sending email requires nearly no resources.

With LLM-training scraper bots, on the other hand, the symptom is the resource consumption. By the time you see their traffic to try to filter it, it's already killing your server. The best you can hope to do is recognize their traffic after a few seconds of firehose and block the IP address.

Then they switch to another one. You block that. They switch to another one.

Residential IPs. Purchased botnet IPs. Constantly rotating IPs.

Unlike spam, there's no reliable way to block an LLM bot that you haven't seen yet, because the only thing that tells you it's a bot is their existing pattern of behavior. And the only unique identifier you can get for them is their IP address.

So how, exactly, are we supposed to filter them effectively, while also allowing legitimate users to access our sites? Especially small-time sites that don't make any money, and thus can't afford to buy CloudFlare or similar protection?

Post reply on HN