Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

61–70 of 279 posts

Re: We can't have nice things because of AI scrapers

#61

Post by tclancy »

blell wrote: post #51
tommek4077 wrote: post #43

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

In these discussions no one will admit this, but the answer is generally yes. Websites written in python and stuff like that.

Yes yes, definitely people don’t know what they’re doing and not that they’re operating on a scale or problem you are not. Metabrainz cannot cache all of these links as most of them are hardly ever hit. Try to assume good intent.

Re: We can't have nice things because of AI scrapers

#62

Post by squigz »

randyl wrote: post #57

The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…

[flagged]

Re: We can't have nice things because of AI scrapers

#63

Post by TeMPOraL »

squigz wrote: post #32
fartfeatures wrote: post #22

Earlier quoted context omitted.

> They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.

The mechanism is putting some text that points to the downloads.

So perhaps it's time to standardize that.

Re: We can't have nice things because of AI scrapers

#65

Post by jakelazaroff »

VTimofeenko wrote: post #58
jacksnipe wrote: post #30

Earlier quoted context omitted.

It’s not great, but you could add it to the body of a 429 response.

Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself

Sometimes! The server can also send a retry-after header to indicate when the client is allowed to request the resource again: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

Re: We can't have nice things because of AI scrapers

#66

Post by zimpenfish »

zzzeek wrote: post #47
rester324 wrote: post #42

Earlier quoted context omitted.

You can implement this yourself, who is stopping you?

Citation needed

I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.)

[0] https://iocaine.madhouse-project.org

Re: We can't have nice things because of AI scrapers

#67

Post by stinky613 »

tommek4077 wrote: post #43

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

A year or two ago I personally encountered scraping bots that were scraping every possible resultant page from a given starting point. So if it scraped a search results page it would also scrape every single distinct combination of facets on that search (including nonsensical combinations e.g. products that match the filter "products where weight2lbs")

We ended up having to block entire ASNs and several subnets (lots from Facebook IPs, interestingly)

Re: We can't have nice things because of AI scrapers

#70

Post by LastTrain »

squigz wrote: post #62
randyl wrote: post #57

The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…

[flagged]

OK. How about shitty and selfish then?
Post reply on HN