I sysadmin my kids' PTA website. OpenAI was scraping it recently. I saw it looking at the event calendar, request after request to random days. I saw years 1000 through 3000 scroll by. I changed the response to their user agent to an access denied, but it still took about 4 hours for them to stop.
We can't have nice things because of AI scrapers
181–190 of 279 posts
Re: We can't have nice things because of AI scrapers
#182Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.
Savvy move by cloudflare, once they have enough sites behind their service they can charge the AI companies to access their cached copies on a back channel
Re: We can't have nice things because of AI scrapers
#183I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent st…
The third-party hit-counting service I use implies that I'm not getting any of this bot scraping on my GitHub blog.
Is Microsoft doing something to prevent it? Or am I so uncool that even bots don't want to read my content :(
Re: We can't have nice things because of AI scrapers
#184I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.
> residential IP pools So, is this a new profit center for sleazeball household ISPs?
No, this is done by paying app developers to bundle some random SDK. Search for Bright Data.
Re: We can't have nice things because of AI scrapers
#185Earlier quoted context omitted.
It’s not great, but you could add it to the body of a 429 response.
Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself
If I find something useful there, I'll read it and code for it...
Re: We can't have nice things because of AI scrapers
#186Earlier quoted context omitted.
And many proxy/scraping providers now use real browsers that can automatically bypass cloudflare captchas as well, and a bot with a real browser similarly won't be clicking on any invisible links, so... I am skeptical just how long this will make an appreciable difference.
So what you’re saying is that a CAPTCHA cannot actually T C and H A.
Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limiting for the rest.
Re: We can't have nice things because of AI scrapers
#187Earlier quoted context omitted.
You do realize that these AI scrapers are most likely written by people who have no idea what they're doing right? Or they just don't care? If they were, pretty much none of the problems these things have caused would exist. Even if we did standardize such a thing, I doubt they would follow it. After all, they think they and everyone else has infinite resources so they can just hammer websites forever.
I realise you are making assertions for which you have no evidence. Until a standard exists we can't just assume nobody will use it, particularly when it makes the very task they are scraping for simpler and more efficient.
> I realise you are making assertions for which you have no evidence.
We do have evidence, which is their current behavior. If they are happy ignoring robots.txt (and also ignoring copyright law), what gives you the belief that they magically won't ignore this new standard? Sure, it in theory might save them money, but if there's one thing that I think is blatantly obvious it is that money isn't what these companies care about because people just keep turning on the money generator. If they did care about it, they wouldn't be spending far more than they earn, and they wouldn't be creating circular economies to try to justify their existences. If my assertion has no evidence, I don't exactly see how yours does either, especially since we have seen that these companies will do anything if it means getting what they want.
Re: We can't have nice things because of AI scrapers
#188Earlier quoted context omitted.
So what you’re saying is that a CAPTCHA cannot actually T C and H A.
Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limitin…
> was never the purpose of CAPTCHAs,
TCHA of CAPTCHA is literally "tell computer human apart"
Re: We can't have nice things because of AI scrapers
#189Earlier quoted context omitted.
Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limitin…
> was never the purpose of CAPTCHAs, TCHA of CAPTCHA is literally "tell computer human apart"
You ignored the emphasis and the rest of the sentence. And the rest of the comment.
(Also, the T was nominally Turing rather than telling.)
Re: We can't have nice things because of AI scrapers
#190Earlier quoted context omitted.
You expect the developers of a crawler to look at every site they crawl and develop a specialized crawler for them? That’s fine if you’re only crawling a handful of sites, but absolutely insane if you’re crawling the entire web.
Isn't the point of AI that it's good at understanding content written for humans? Why can't the scrapers run the homepage through an LLM to detect that? I'm also not sure why we should be prioritizing the needs of scraper writers over human users and site operators.
How is passing a site's homepage to an LLM supposed to make it develop a custom crawler?