Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.
Wikipedia is struggling with voracious AI bot crawlers
31–40 of 105 posts
Re: Wikipedia is struggling with voracious AI bot crawlers
#32I thought all of Wikipedia can be downloaded directly if that's the goal? [0] Why scrape? [0] https://en.wikipedia.org/wiki/Wikipedia:Database_download
Someone's gotta tell the LLMs that when a prompt-kiddie asks them to build a scraper bot that "I suggest downloading the database instead".
Re: Wikipedia is struggling with voracious AI bot crawlers
#33People got to make bots pay. That's the only way to get rid of this world wide DDOSing backed up by multi billions companies. There are captcha to block bots or at least make them pay money to solve them, some people in Linux community also made tools to combat that, i think something that use a little cpu energy. And in the same time, you offer an api, less expensive than the cost to crawl it, and everyone win. Mult…
This is an interesting model in general: free for humans, pay for automation. How do you enforce that though? Captchas sounds like a waste.
Re: Wikipedia is struggling with voracious AI bot crawlers
#34This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
I've written some unfathomably bad web crawlers in the past. Indeed, web crawlers might be the most natural magnet for bad coding and eye-twitchingly questionable architectural practices I know of. While it likely isn't the major factor here I can attest that there are coders who see pages-articles-multistream.xml.bz2 and then reach for a wget + HTML parser combo. If you don't live and breath Wikipedia it is going to…
You'd probably ask ChatGPT to write you a crawler for Wikipedia, without thinking to ask whether there's a better way to get Wikipedia info. So that download would be missed, because how and what we ask AI stays very important. Actually this is not new, googling skills were known as being important before and even philosophers recognized that asking good questions was crucial.
Re: Wikipedia is struggling with voracious AI bot crawlers
#35This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
I've written some unfathomably bad web crawlers in the past. Indeed, web crawlers might be the most natural magnet for bad coding and eye-twitchingly questionable architectural practices I know of. While it likely isn't the major factor here I can attest that there are coders who see pages-articles-multistream.xml.bz2 and then reach for a wget + HTML parser combo. If you don't live and breath Wikipedia it is going to…
You don't even need to deal with any XML formats or anything, they publish a complete dataset on Huggingface that's just a few lines to load in your Python training script
Re: Wikipedia is struggling with voracious AI bot crawlers
#36I thought all of Wikipedia can be downloaded directly if that's the goal? [0] Why scrape? [0] https://en.wikipedia.org/wiki/Wikipedia:Database_download
Turns out AI isn't smart enough to figure this out yet.
Re: Wikipedia is struggling with voracious AI bot crawlers
#37I was on a panel with the President of Wikimedia LLC at SXSW and this was brought up. There's audio attached https://schedule.sxsw.com/2025/events/PP153044.
I also like Anna's (Creative Commons) framing of the problem being money + attribution + reciprocity.
Re: Wikipedia is struggling with voracious AI bot crawlers
#38what's the best way to stop the bots? cloudflare?
Re: Wikipedia is struggling with voracious AI bot crawlers
#39Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.
That was always my feeling - however big Wikipedia is, however many requests it handles, it is not big enough to warrant burning so many millions every year. Alas, I never studied the subject, as overall I feel Wikipedia is good force in the Universe.
So perhaps price comes with the greatness?
Re: Wikipedia is struggling with voracious AI bot crawlers
#40Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.
BTW, I do not know why you are getting downvoted, this is a real concern that someone needs to tackle one day.