Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia is struggling with voracious AI bot crawlers

engadget.com

Post reply on HN
Print view View original

21–30 of 105 posts

Re: Wikipedia is struggling with voracious AI bot crawlers

#21

Post by nonrandomstring »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

> Why would you crawl the web interface when the data is so readily available in a even better format?

To cause deliberate harm as a DDOS attack. Perhaps a better question is, why would companies who hope to replace human-curated static online information with their own generative service not use the cloak of "scraping" to take down their competition?

Re: Wikipedia is struggling with voracious AI bot crawlers

#22

Post by roenxi »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

I've written some unfathomably bad web crawlers in the past. Indeed, web crawlers might be the most natural magnet for bad coding and eye-twitchingly questionable architectural practices I know of. While it likely isn't the major factor here I can attest that there are coders who see pages-articles-multistream.xml.bz2 and then reach for a wget + HTML parser combo.

If you don't live and breath Wikipedia it is going to soak up a lot of time figuring out Wikipedia's XML format and markup language, not to mention re-learning how to parse XML. HTTP requests and bashing through the HTML is all everyday web skills and familiar scripting that is more reflexive and well understood. The right way would probably be much easier but figuring it out will take too long.

Although that is all pre-ChatGPT logic. Now I'd start by asking it to solve my problem.

Re: Wikipedia is struggling with voracious AI bot crawlers

#23

Post by is_true »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

This is what you get when an AI generates your code and your prompts are vague.

Re: Wikipedia is struggling with voracious AI bot crawlers

#24

Post by Cthulhu_ »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

Because the scrapers they use aren't targeted, they just try to index the whole internet. It's easier that way.

Re: Wikipedia is struggling with voracious AI bot crawlers

#25

Post by spiderfarmer »

lambdaone wrote: post #19

It's not just Wikipedia - the entire rest of the open-access web is suffering with them. I think the most interesting thing here is that it shows that the companies doing these crawls simply don't care who they hurt, as they actively take measures to prevent their victims from stopping them by using multiple IP addresses, snowshoe crawling, evading fingerprinting, and so on. For Wikipedia, there's a solution served u…

Truth. I have a platform with over a million photos. It costs me a lot of bandwidth.

Re: Wikipedia is struggling with voracious AI bot crawlers

#26

Post by MiscIdeaMaker99 »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

> Why would you crawl the web interface when the data is so readily available in a even better format?

It's entirely possible they don't know about this. I certainly didn't until just now.

Re: Wikipedia is struggling with voracious AI bot crawlers

#27

Post by aucisson_masque »

People got to make bots pay. That's the only way to get rid of this world wide DDOSing backed up by multi billions companies.

There are captcha to block bots or at least make them pay money to solve them, some people in Linux community also made tools to combat that, i think something that use a little cpu energy.

And in the same time, you offer an api, less expensive than the cost to crawl it, and everyone win.

Multi billions companies get their sweet sweet data, Wikipedia gets money to enhance their infrastructure or whatever, users benefits from Wikipedia quality engagement.

Re: Wikipedia is struggling with voracious AI bot crawlers

#28

Post by iamacyborg »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

With the way Transclusion works in MediaWiki, dumps and the wiki api’s are often not very useful, unfortunately

Re: Wikipedia is struggling with voracious AI bot crawlers

#30

Post by coldpie »

latexr wrote: post #18
diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

> Why would you crawl the web interface when the data is so readily available in a even better format? Because grifters have no respect or care for other people, nor are they interested in learning how to be efficient. They only care about the least amount of effort for the largest amount of personal profit. Why special-case Wikipedia, when they can just scratch their balls and turn their code loose? It’s not their o…

The vast, vast majority of companies using AI are on the same level as the people distributing malware to mine crypto on other peoples' machines. They're exploiting resources that aren't theirs to get rich quick from stupid investors & market hype. We all suffer so they can get a couple bucks. Thanks, AI & braindead investors. This bubble can't pop soon enough, and I hope it takes a whole lot of terrible people down with it.
Post reply on HN