"async_visit_link for link in links omg it works"
Wikipedia is struggling with voracious AI bot crawlers
41–50 of 105 posts
Re: Wikipedia is struggling with voracious AI bot crawlers
#42This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…
I've written some unfathomably bad web crawlers in the past. Indeed, web crawlers might be the most natural magnet for bad coding and eye-twitchingly questionable architectural practices I know of. While it likely isn't the major factor here I can attest that there are coders who see pages-articles-multistream.xml.bz2 and then reach for a wget + HTML parser combo. If you don't live and breath Wikipedia it is going to…
To be a "good" web crawler, you have to go beyond "not bad coding". If you just write the natural "fetch page, fetch next page, retry if it fails" loop, notably, missing any sort of wait between fetches, so that you fetch as quickly as possible, you are already a pest. You don't even need multiple threads or machines to be a pest; a single machine on a home connection fetching pages as quickly as it can be already be a pest to a website with heavy backend computation or DB demands. Do an equally naive "run on a couple dozen threads" upgrade to your code and you expand the blast radius of your pestilence out to even more web sites.
Being a truly good web crawler takes a lot of work, and being a polite web crawler takes yet more different work.
And then, of course, you add the bad coding practices on top of it, ignoring robots.txt or using robots.txt as a list of URLs to scrape (which can be either deliberate or accidental), hammering the same pages over and over, preferentially "retrying" the very pages that are timing out because you found the page that locks the DB for 30 seconds in a hard query that even the website owners themselves didn't know was possible until you showed them by taking down the rest of their site in the process... it just goes downhill from there. Being "not bad" is already not good enough and there's plenty of "bad" out there.
Re: Wikipedia is struggling with voracious AI bot crawlers
#43Earlier quoted context omitted.
That is not an excuse. Wikipedia isn't just any site.
Not an excuse, a plausible explanation of what's actually happening.
Also plausibly they are trying to kill the site via soft ddos. Then they can sell a service based on all the data they scraped + unauditable censoring.
Re: Wikipedia is struggling with voracious AI bot crawlers
#44I thought all of Wikipedia can be downloaded directly if that's the goal? [0] Why scrape? [0] https://en.wikipedia.org/wiki/Wikipedia:Database_download
Someone's gotta tell the LLMs that when a prompt-kiddie asks them to build a scraper bot that "I suggest downloading the database instead".
This is the first time I'm reading "prompt-kiddie", made me chuckle hard. Jumped straight into my vocabulary :-)
Re: Wikipedia is struggling with voracious AI bot crawlers
#45what's the best way to stop the bots? cloudflare?
make pain for the people they serve.
Re: Wikipedia is struggling with voracious AI bot crawlers
#46[dead]
Re: Wikipedia is struggling with voracious AI bot crawlers
#47People got to make bots pay. That's the only way to get rid of this world wide DDOSing backed up by multi billions companies. There are captcha to block bots or at least make them pay money to solve them, some people in Linux community also made tools to combat that, i think something that use a little cpu energy. And in the same time, you offer an api, less expensive than the cost to crawl it, and everyone win. Mult…
This is an interesting model in general: free for humans, pay for automation. How do you enforce that though? Captchas sounds like a waste.
Any plan that starts with "Step one: Apply the tool that almost perfectly distinguishes human traffic from non-human traffic" is doomed to failure. That's whatever the engineering equivalent of "begging the question" is, where the solution to the problem is that we assume that we have the solution to the problem.
Re: Wikipedia is struggling with voracious AI bot crawlers
#48People got to make bots pay. That's the only way to get rid of this world wide DDOSing backed up by multi billions companies. There are captcha to block bots or at least make them pay money to solve them, some people in Linux community also made tools to combat that, i think something that use a little cpu energy. And in the same time, you offer an api, less expensive than the cost to crawl it, and everyone win. Mult…
This is an interesting model in general: free for humans, pay for automation. How do you enforce that though? Captchas sounds like a waste.
Why not just rate limit every user to realistic human rates. You just punish anyone behaving like a bot.
Re: Wikipedia is struggling with voracious AI bot crawlers
#49what's the best way to stop the bots? cloudflare?
Why should we stop the bots? Wikipedia supposedly wants the world to have this free information, a bit is just another way of supporting that goal.
Re: Wikipedia is struggling with voracious AI bot crawlers
#50what's the best way to stop the bots? cloudflare?
my guess is the gnome anime girl anti bot captcha