Wikipedia is struggling with voracious AI bot crawlers
1–10 of 105 posts
Re: Wikipedia is struggling with voracious AI bot crawlers
#2Wouldn't downloading the publicly available Wikipedia database (e.g. via Torrent [1]) be enough for AI training purposes? I get that this doesn't actually stop AI bots, but captchas and other restrictions would undermine the open nature of Wikipedia.
[1] https://en.wikipedia.org/wiki/Wikipedia:Database_download
Re: Wikipedia is struggling with voracious AI bot crawlers
#3Wouldn't downloading the publicly available Wikipedia database (e.g. via Torrent [1]) be enough for AI training purposes? I get that this doesn't actually stop AI bots, but captchas and other restrictions would undermine the open nature of Wikipedia. [1] https://en.wikipedia.org/wiki/Wikipedia:Database_download
[deleted]
Re: Wikipedia is struggling with voracious AI bot crawlers
#4This has to be one of strangest targets to crawl, since they themselves make database dumps available for download (https://en.wikipedia.org/wiki/Wikipedia:Database_download) and if that wasn't enough, there are 3rd party dumps as well (https://library.kiwix.org/#lang=eng&category=wikipedia) that you could use if the official ones aren't good enough for some reason.
Why would you crawl the web interface when the data is so readily available in a even better format?
Re: Wikipedia is struggling with voracious AI bot crawlers
#5Wikipedia provides dumps. Probably cheaper and easier than crawling it. Given the size of Wikipedia it would be well worth a little extra code. it also avoids the risk of getting blocked, and is more reliable.
It suggest to me that people running AI crawlers are throwing resources at the problem with little thought.
Re: Wikipedia is struggling with voracious AI bot crawlers
#6I thought all of Wikipedia can be downloaded directly if that's the goal? [0] Why scrape?
[0] https://en.wikipedia.org/wiki/Wikipedia:Database_download
Re: Wikipedia is struggling with voracious AI bot crawlers
#7The worst thing about that is that wikipedia has dumps of all its data which you can download.
Re: Wikipedia is struggling with voracious AI bot crawlers
#8This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
I think those crawlers are just very generic: they basically operate like wget scripts, without much logic for avoiding sites that already offer clean data dumps.
Re: Wikipedia is struggling with voracious AI bot crawlers
#9Wikipedia provides dumps. Probably cheaper and easier than crawling it. Given the size of Wikipedia it would be well worth a little extra code. it also avoids the risk of getting blocked, and is more reliable. It suggest to me that people running AI crawlers are throwing resources at the problem with little thought.
Maybe they just vibe-coded the crawlers and that's why they don't work very well or know the best way to do it
Re: Wikipedia is struggling with voracious AI bot crawlers
#10This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
I think most of these crawlers just aren't very well implemented. Takes a lot of time and effort to get crawling to work well, very easy to accidentally DoS a website if you don't pay attention.