Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

111–120 of 279 posts

Re: We can't have nice things because of AI scrapers

#111

Post by TeMPOraL »

squigz wrote: post #79
TeMPOraL wrote: post #63

Earlier quoted context omitted.

So perhaps it's time to standardize that .

I'm not entirely sure why people think more standards are the way forward. The scrapers apparently don't listen to the already-established standards. What makes one think they would suddenly start if we add another one or two?

There is no standard, well-known way for a website to advertise, "hey, here's a cached data dump for bulk download, please use that instead of bulk scraping". If they were, I'd expect the major AI companies and other users[0] to use that method for gathering training data[1]. They have compelling reasons to: it's cheaper for them, and cultivates goodwill instead of burning it.

This also means that right now, it could be much easier to push through such standard than ever before: there are big players who would actually be receptive to it, so even few not-entirely-selfish actors agreeing on it might just do the trick.

--

[0] - Plenty of them exist. Scrapping wasn't popularized by AI companies, it's standard practice of on-line business in competitive markets. It's the digital equivalent of sending your employees to competing stores undercover.

[1] - Not to be confused with having an LLM scrap specific page for some user because the user requested it. That IMO is a totally legitimate and unfairly penalized/villified use case, because LLM is acting for the user - i.e. it becomes a literal user agent, in the same sense that web browser is (this is the meaning behind the name of "User-Agent" header).

Re: We can't have nice things because of AI scrapers

#112

Post by tommek4077 »

switz wrote: post #55
tommek4077 wrote: post #43

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

If you have a lot of pages, AI bots will scrape every single one on a loop - wiki's generally don't have anywhere near the number of pages as an incremented entity primary id. I have a few million pages on a tiny website and it gets hammered by AI bots all day long. I can handle it, but it's a nuisance and they're basically just scraping garbage (statistics pages of historical matches or user pages that have essentia…

How many requests per second do you get? I also see a lot of bot traffic but nowhere near to hit the servers significantly, and i render most stuff on the server directly.

Re: We can't have nice things because of AI scrapers

#113

Post by lysace »

Forgeties79 wrote: post #110
lysace wrote: post #78

Earlier quoted context omitted.

[flagged]

There are people behind the web crawler. If they’re so well funded they can exert a little effort to not so badly inconvenience people as they steal their training data.

[flagged]

Re: We can't have nice things because of AI scrapers

#114

Post by gamer191 »

accrual wrote: post #101

Grateful for Metabrainz putting in this work to keep the service up. We really ought to have some kind of "I am an AI!" signal to route the request properly into a long queue...

Companies wouldn’t send it because they know that most websites would block them

Re: We can't have nice things because of AI scrapers

#115

Post by doublerabbit »

zimpenfish wrote: post #66
zzzeek wrote: post #47

Earlier quoted context omitted.

Citation needed

I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org

Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth.

I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior.

I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?

Re: We can't have nice things because of AI scrapers

#116

Post by chao- »

zimpenfish wrote: post #66
zzzeek wrote: post #47

Earlier quoted context omitted.

Citation needed

I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org

Can't seem to access this.

It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux?

EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?

Re: We can't have nice things because of AI scrapers

#117

Post by acdha »

rudedogg wrote: post #87
timpera wrote: post #31

Earlier quoted context omitted.

Unfortunately, Cloudflare often destroys the experience for users with shared connections, VPNs, exotic browsers… I had to remove it from my site after too many complaints.

Also iCloud Private Relay. CloudFlare is making it impossible to browse privately

Cloudflare works fine with public relay - they and Fastly provide infrastructure for that service (one half of the blinded pair) so it’s definitely something they test.

Re: We can't have nice things because of AI scrapers

#118

Post by VonGallifrey »

TeMPOraL wrote: post #95
tooltower wrote: post #80

> Rather than downloading our dataset in one complete download, they insist on loading all of MusicBrainz one page at a time. Is there a standard mechanism for batch-downloading a public site? I'm not too familiar with crawlers these days.

There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience. Anyway, all that means there was never a critic…

> turns out plenty of people say they write for others to read

LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.

Re: We can't have nice things because of AI scrapers

#119

Post by chao- »

stinky613 wrote: post #67
tommek4077 wrote: post #43

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

A year or two ago I personally encountered scraping bots that were scraping every possible resultant page from a given starting point. So if it scraped a search results page it would also scrape every single distinct combination of facets on that search (including nonsensical combinations e.g. products that match the filter "products where weight 2lbs") We ended up having to block entire ASNs and several subnets (lot…

I have encountered this same issue with faceted search results and individual inventory listings.

Re: We can't have nice things because of AI scrapers

#120

Post by consp »

PrairieFire wrote: post #84
squigz wrote: post #62

Earlier quoted context omitted.

[flagged]

I am with you in that this rhetoric is getting exhausting. In this particular case though I don't think "evil” is a moral claim, more shorthand for cost externalizing behavior. Hammering expensive dynamic endpoints with millions of unique requests isn’t neutral automation, it's degrading a shared public resource. Call it evil, antisocial, or extractive, the outcome is the same.

> shorthand for cost externalizing behavior

I consider that evil, having no regard for the wellbeing of others for you own greed.

Post reply on HN