Grateful for Metabrainz putting in this work to keep the service up. We really ought to have some kind of "I am an AI!" signal to route the request properly into a long queue...
We can't have nice things because of AI scrapers
101–110 of 279 posts
Re: We can't have nice things because of AI scrapers
#102random idea, instead of blocking scrapers, maybe detect them (via user-agent, request patterns, ignoring robots.txt) and serve them garbage data wrapped in dad jokes.
if (isSuspiciousScraper(req)) {
return res.json({
data: getDadJoke(),
artist: "Rick Astley", // always
album: "Never Gonna Give You Up"
});
}Re: We can't have nice things because of AI scrapers
#103Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…
Yeah AI scrapers is one of the reasons why i have closed my public website https://tvnfo.com and only left donors site online. It’s not only because of AI scrapers but i grew tired of people trying to scrape the site eating a lot of reasorcers this small project don’t have. Very sad really it was publicly online since 2016. Now it’s only available for donors. Running a tiny project on just $60 a month. If this was not my hobby i would close it completely long time ago :-) Who know if there is more support in the future i might reopen public site again with something like anubes bot protection. But i thought it was only small sites like mine who gets hit hard, looks like many have similar issues. Soon nothing will be open or useful online. I wonder if this was the plan all along whoever pushing AI on massive scale.
Re: We can't have nice things because of AI scrapers
#104the more time passes the more i'm convinced that the solution is to—somehow—force everyone to have to go through something like common crawl i don't want people's servers to be pegged at 100% because a stupid dfs scraper is exhaustively traversing their search facets, but i also want the web to remain scrapable by ordinary people, or rather go back to how readily scrapable it used to be before the invention of cloudf…
Seriously, I can't help but think this has to be part of the answer.
Just something like /llms.txt which contains a list of .txt or .txt.gz files or something?
Because the problem is that every site is going to have its own data dump format, often in complex XML or SQL or something.
LLM's don't need any of that metadata, and many sites might not want to provide it because e.g. Yelp doesn't want competitors scraping its list of restaurants.
But if it's intentionally limited to only paragraph-style text, and stripped entirely of URL's, ID's, addresses, phone numbers, etc. -- so e.g. a Yelp page would literally just be the cuisine category and reviews of each restaurant, no name, no city, no identifier or anything -- then it gives LLM's what they need much faster, the site doesn't need to be hammered, and it's not in a format for competitors to easily copy your content.
At most, maybe add markup for to represent pages, products, restaurants, whatever the "main noun" is, and recursive to represent e.g. reviews on a restaurant, comments on a review, comments one level deeper on a comment, etc. Maybe a couple more like and , but otherwise just pure text. As simple as possible.
The biggest problem is that a lot of sites will create a "dummy" llms.txt without most of the content because they don't care, so the scrapers will scrape anyways...
Re: We can't have nice things because of AI scrapers
#105The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…
[flagged]
Absolutely evil.
Re: We can't have nice things because of AI scrapers
#106Its not just AI scrappers doing it by themselves but now users are also being trained to put the link in the claude chat/chat gpt and ask it to summarise it. And off course that would show up on the website end as a scraper. In fact firefox now allows you to preview the link and get key points without ever going to the link[1] [1] https://imgur.com/a/3E17Dts
It's three issues:
- AI shops scraping the web to update their datasets without respecting netiquette (or sometimes being unable to automate it for every site due to the scale, ironically).
- People extensively using agents (search, summarizers, autonomous agents etc), which are indistinguishable from scraper bots from website's perspective.
- Agents being both faster and less efficient (more requests per action) than humans.
Re: We can't have nice things because of AI scrapers
#107it's funny how ai is the problem that the cryptocurrencyverse was always in search of...
Re: We can't have nice things because of AI scrapers
#108Earlier quoted context omitted.
In these discussions no one will admit this, but the answer is generally yes. Websites written in python and stuff like that.
Yes yes, definitely people don’t know what they’re doing and not that they’re operating on a scale or problem you are not. Metabrainz cannot cache all of these links as most of them are hardly ever hit. Try to assume good intent.
But serving HTML is unbelievably cheap, isn't it?
Re: We can't have nice things because of AI scrapers
#109Nowadays people complain about AI scrapers with the same vain as they complained about search indexers a way back when. Just a few years later, people had stopped caring too much about storage access and bandwidth, and started begging search engines to visit their websites. Every trick on the planet Earth, SEO optimization, etc. Looking forward to the time when everybody suddenly starts to embrace AI indexers and wel…
We already know the solution: One well-behaved, shared scraper could serve all of the AI companies simultaneously.
The problem is that they're not doing it.
Re: We can't have nice things because of AI scrapers
#110The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…
[flagged]
There are people behind the web crawler. If they’re so well funded they can exert a little effort to not so badly inconvenience people as they steal their training data.