Earlier quoted context omitted.
But why?
No humans, no point.
But AI scraping doesn't remove humans...?
Even if humans make up a smaller proportion of your traffic, they're still the same number in absolute terms.
We can't have nice things because of AI scrapers
191–200 of 279 posts
Re: We can't have nice things because of AI scrapers
#192Nowadays people complain about AI scrapers with the same vain as they complained about search indexers a way back when. Just a few years later, people had stopped caring too much about storage access and bandwidth, and started begging search engines to visit their websites. Every trick on the planet Earth, SEO optimization, etc. Looking forward to the time when everybody suddenly starts to embrace AI indexers and wel…
Bad take. Search engines send people to your site, LLMs don’t.
I visit sites and pages through links I get from an LLM plenty.
Re: We can't have nice things because of AI scrapers
#193there's this one guy I know who writes scripts to poison ai scrapers that ignore the robot.txt, basically creates word salad and random folders that keep going deep without giving the ai anything of value.
Re: We can't have nice things because of AI scrapers
#194Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…
> Or better still, this torrent file, where the bots would briefly end up improving the shareability of the data.
Depends on if they wrote their own BitTorrent client or not. It’s possible to write a client that doesn’t share, and even reports false/inflated sharing stats back to the tracker.
A decade or more ago I modified my client to inflate my share stats so I wouldn’t get kicked out of a private tracker whose high share ratios conflicted with my crappy data plan.
Re: We can't have nice things because of AI scrapers
#195I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent st…
And those are the good actors! We're under a crawlocalpyse from botnets, er, residential proxies. "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", anyone?
Yeah the flood of these Chrome UAs with every version number under the sun, and a really large portion being *.0.0.0 version numbers, that's what I've tended to experience lately. Also just kind of every browser user agent ever:
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; .NET CLR 3.5.21022)
There were waves of big and sometimes intrusive traffic admitting to being from Amazon, Anthropic, Google, Meta, etc., but those are easy to block or throttle and aren't that big a deal in the scheme of things.
Re: We can't have nice things because of AI scrapers
#196Earlier quoted context omitted.
> To archive Metabrainz there is no way but to browse the pages slowly page-by-page. There's no machine-communicable way that suggests an alternative. Why does there have to be a "machine-communicable way"? If these developers cared about such things they would spend 20 seconds looking at this page. It's literally one of the first links when you Google "metabrainz" https://metabrainz.org/datasets
You expect the developers of a crawler to look at every site they crawl and develop a specialized crawler for them? That’s fine if you’re only crawling a handful of sites, but absolutely insane if you’re crawling the entire web.
if you are crawling the entire web, you should respect robots.txt and don't fetch anything disallowed. full stop.
Re: We can't have nice things because of AI scrapers
#197How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
The worse thing is calendar/schedule. Many crawler tries to load every single day, with day view, week view and month view.
Those pages are dynamically generated and virtually limitless
Re: We can't have nice things because of AI scrapers
#198Grateful for Metabrainz putting in this work to keep the service up. We really ought to have some kind of "I am an AI!" signal to route the request properly into a long queue...
Yeah, they can just set the evil bit in their IP packets.
Re: We can't have nice things because of AI scrapers
#199I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent st…
The third-party hit-counting service I use implies that I'm not getting any of this bot scraping on my GitHub blog. Is Microsoft doing something to prevent it? Or am I so uncool that even bots don't want to read my content :(
I'm interested in that service and how it works. Link?
Re: We can't have nice things because of AI scrapers
#200True. I had to kill my dynamic service of collected film festival ratings, because the bots drained the memory of the still available free hosters. I fought it for two years, with user agent and IP ranges, but eventually gave up. So I had to revert to static pages hosted on GitHub pages. The bots cannot kill that. But very limited features