Maybe the big tech providers should play fair and host the downloadable database for those bots as well as crawlable mirrors.
Wikipedia is struggling with voracious AI bot crawlers
11–20 of 105 posts
Re: Wikipedia is struggling with voracious AI bot crawlers
#12This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
I think those crawlers are just very generic: they basically operate like wget scripts, without much logic for avoiding sites that already offer clean data dumps.
That is not an excuse. Wikipedia isn't just any site.
Re: Wikipedia is struggling with voracious AI bot crawlers
#13Earlier quoted context omitted.
I think those crawlers are just very generic: they basically operate like wget scripts, without much logic for avoiding sites that already offer clean data dumps.
That is not an excuse. Wikipedia isn't just any site.
Not an excuse, a plausible explanation of what's actually happening.
Re: Wikipedia is struggling with voracious AI bot crawlers
#14Not just Wikipedia. My home server (hosting a number of not particularly noteworthy things, such as my personal gitea instance) has been absolutely hammered in recent months, to the extent of periodically bringing down the server for hours with thrashing.
The worst part is that every single sociopathic company in the world seems to have simultaneously unleashed their own fleet of crawlers.
Most of the bots downright ignore robots.txt, and some of the crawlers hit the site simultaneously from several IPs. I've been trying to lure the bots into a nepenthes tarpit, which somewhat helps, but ultimately find myself having to firewall entire IP ranges.
Re: Wikipedia is struggling with voracious AI bot crawlers
#15Wikipedia provides dumps. Probably cheaper and easier than crawling it. Given the size of Wikipedia it would be well worth a little extra code. it also avoids the risk of getting blocked, and is more reliable. It suggest to me that people running AI crawlers are throwing resources at the problem with little thought.
Maybe they just vibe-coded the crawlers and that's why they don't work very well or know the best way to do it
Maybe they should just ... not "vibe-code" at all then ?
Re: Wikipedia is struggling with voracious AI bot crawlers
#16IPFS
Re: Wikipedia is struggling with voracious AI bot crawlers
#17Re: Wikipedia is struggling with voracious AI bot crawlers
#18This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
> Why would you crawl the web interface when the data is so readily available in a even better format?
Because grifters have no respect or care for other people, nor are they interested in learning how to be efficient. They only care about the least amount of effort for the largest amount of personal profit. Why special-case Wikipedia, when they can just scratch their balls and turn their code loose? It’s not their own money they’re burning anyway; there are more chumps throwing money at them than they know what to do with, so it’s imperative they look competitive and hard at work.
Re: Wikipedia is struggling with voracious AI bot crawlers
#19It's not just Wikipedia - the entire rest of the open-access web is suffering with them.
I think the most interesting thing here is that it shows that the companies doing these crawls simply don't care who they hurt, as they actively take measures to prevent their victims from stopping them by using multiple IP addresses, snowshoe crawling, evading fingerprinting, and so on.
For Wikipedia, there's a solution served up to them on a plate. But they simply can't be bothered to take it.
And this in turn shows the overall moral standards of those companies - it's the wild west out there, where the weak go to the wall, and those inflicting the damage know what they're doing, and just don't care. Sociopaths.
Re: Wikipedia is struggling with voracious AI bot crawlers
#20This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
There are crawlers that will recursively crawl source repository web interfaces (cgit et al, usually expensive to render) despite having a readily available URL they could clone from. At this point I'm not far from assuming malice over sheer incompetence.