Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia is struggling with voracious AI bot crawlers

engadget.com

Post reply on HN
Print view View original

51–60 of 105 posts

Re: Wikipedia is struggling with voracious AI bot crawlers

#51

Post by skywhopper »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

Because AI and everything about it is about being as lazy as possible and wasting money and compute in service of becoming even lazier. No one who is willing to burn the compute necessary to train the big models we see would think twice about the wasted resources involved in doing the most wasteful, least efficient means of collecting the data possible.

Re: Wikipedia is struggling with voracious AI bot crawlers

#52

Post by delichon »

We're having the same trouble for a few hundred sites that we manage. It is no problem for crawlers that obey robots.txt since we ask for one visit per 10 seconds, which is manageable. The problem seems to be mostly the greedy bots that request as fast as we can reply. So my current plan is to set rate limiting for everyone, bots or not. But doing stats on the logs, it isn't easy to figure out a limit that won't bounce legit human visitors.

The bigger problem is that the LLMs are so good that their users no longer feel the need to visit these sites directly. It looks like the business model of most of our clients is becoming obsolete. My paycheck is downstream of that, and I don't see a fix for it.

Re: Wikipedia is struggling with voracious AI bot crawlers

#54

Post by coldpie »

briandear wrote: post #49
amazingamazing wrote: post #38

what's the best way to stop the bots? cloudflare?

Why should we stop the bots? Wikipedia supposedly wants the world to have this free information, a bit is just another way of supporting that goal.

The answer to your question is in the article.

Re: Wikipedia is struggling with voracious AI bot crawlers

#55

Post by richiebful1 »

briandear wrote: post #49
amazingamazing wrote: post #38

what's the best way to stop the bots? cloudflare?

Why should we stop the bots? Wikipedia supposedly wants the world to have this free information, a bit is just another way of supporting that goal.

Because there's a better way for bots to get the data via the wikipedia database dump. Sending some large zip archives is a lot cheaper than individually serving every page on Wikipedia.

Re: Wikipedia is struggling with voracious AI bot crawlers

#56

Post by briandear »

limaoscarjuliet wrote: post #40
m101 wrote: post #31

Wikipedia spends 1% of its budget on hosting fees. It can spend a bit more given the rest of their corruptions.

BTW, I do not know why you are getting downvoted, this is a real concern that someone needs to tackle one day.

It’s getting downvoted because the parent comment aligns with what Elon said about Wikipedia; so it’s a knee jerk reaction. Though the sentiment is factual.

Previous discussion: (2022) https://news.ycombinator.com/item?id=32840097

Re: Wikipedia is struggling with voracious AI bot crawlers

#57

Post by guerrilla »

karn97 wrote: post #48
guerrilla wrote: post #33

Earlier quoted context omitted.

This is an interesting model in general: free for humans, pay for automation. How do you enforce that though? Captchas sounds like a waste.

Why not just rate limit every user to realistic human rates. You just punish anyone behaving like a bot.

Sold. Pay by page retrieval rate.

Re: Wikipedia is struggling with voracious AI bot crawlers

#58

Post by shreyshnaccount »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

i was thinking the same thing. it might be the case that the scrapers are result of the web search feature in llms?

Re: Wikipedia is struggling with voracious AI bot crawlers

#59

Post by szszrk »

gherard5555 wrote: post #50
amazingamazing wrote: post #38

what's the best way to stop the bots? cloudflare?

my guess is the gnome anime girl anti bot captcha

mandatory link: https://github.com/TecharoHQ/anubis

It's an interesting project, I wish there would be better ways to do that, but I guess we are on war with crawlers for a while already.

Re: Wikipedia is struggling with voracious AI bot crawlers

#60

Post by cowsaymoo »

is_true wrote: post #23
diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

This is what you get when an AI generates your code and your prompts are vague.

Vibe coded crawlers
Post reply on HN