51–60 of 228 posts
I presume people have logs that indicate the source for them to place blame on AI scrapers. Is anybody making these available for analysis so we can see exactly who is doing this?
The author of this post could solve their problem with Cloudflare or any of its numerous competitors. Cloudflare will even do it for free.
They don't. I'm using Cloudflare and 90%+ of the traffic I'm getting are still broken scrapers, a lot of them coming through residential proxies. I don't know what they block, but they're not very good at that. Or, to be more fair: I think the scrapers have gotten really good at what they do because there's real money to be made.
I presume people have logs that indicate the source for them to place blame on AI scrapers. Is anybody making these available for analysis so we can see exactly who is doing this?
A lot of compromised home devices and cheap servers proxying traffic, from all over the world.
Earlier quoted context omitted.
because he's unable to self-host git anymore because AI bots are hammering it to submit PRs. self-hosting was originally a "right" we had upon gaining access to the internet in the 90s, it was the main point of the hyper text transfer protocol.
sorry if i missed it, but the original post doesn't say anything about PRs... the bots only seem to be scraping the content
Earlier quoted context omitted.
A lot of compromised home devices and cheap servers proxying traffic, from all over the world.
If that is the case how can you determine the reason for the activity?
Here-in is the problem. And if you block them, you risk blocking actual customers.
Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
Can we not charge for access? If I have a link, that says "By clicking this link you agree to pay $10 for each access" then sending the bill?
Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
Right, this is exactly what they are.
They're written by people who a) think they have a right to every piece of data out there, b) don't have time (or shouldn't have to bother spending time) to learn any kind of specifics of any given site and c) don't care what damage they do to anyone else as they get the data they crave.
(a) means that if you have a robots.txt, they will deliberately ignore it, even if it's structured to allow their bots to scrape all the data more efficiently. Even if you have an API, following it would require them to pay attention to your site specifically, so by (b), they will ignore that too—but they also ignore it because they are essentially treating the entire process as an adversarial one, where the people who hold the data are actively trying to hide it from them.
Now, of course, this is all purely based on my observations of their behavior. It is possible that they are, in fact, just dumb as a box of rocks...and also don't care what damage they do. (c) is clearly true regardless of other specific motives.
Can we not charge for access? If I have a link, that says "By clicking this link you agree to pay $10 for each access" then sending the bill?
I have no idea if it actually works as advertised though. I don't think I've heard from anyone trying it.
The Chinese AI scrapers/bots are killing quite a bit of the regular web now. YisouSpider absolutely pummeled my open source project's hosting for weeks. Like all Chinese AI scrapers, it ignores robots.txt. So forget about it respecting a Crawl-delay. If you block the user agent, it would calm down for a bit, then it would just come back again using a generic browser user agent from the same IP addresses. It does this…