Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wikipedia is struggling with voracious AI bot crawlers

engadget.com

Post reply on HN
Print view View original

91–100 of 105 posts

Re: Wikipedia is struggling with voracious AI bot crawlers

#91

Post by limaoscarjuliet »

jraph wrote: post #77
limaoscarjuliet wrote: post #40

Earlier quoted context omitted.

BTW, I do not know why you are getting downvoted, this is a real concern that someone needs to tackle one day.

Real concern or not, this is not related to the discussion at hand, which is AI crawlers hammering Wikipedia, which is related to AI crawlers hammering everything these days. Here's the concern at hand. I would like to read on Wikipedia corruption with quality sources (in a separate HN post, which would probably be successful), but that's not quite on-topic here. Not only it's off-topic and borderline whataboutism, i…

Fair enough!

Re: Wikipedia is struggling with voracious AI bot crawlers

#92

Post by [deleted] »

coldpie wrote: post #30
latexr wrote: post #18

Earlier quoted context omitted.

> Why would you crawl the web interface when the data is so readily available in a even better format? Because grifters have no respect or care for other people, nor are they interested in learning how to be efficient. They only care about the least amount of effort for the largest amount of personal profit. Why special-case Wikipedia, when they can just scratch their balls and turn their code loose? It’s not their o…

The vast, vast majority of companies using AI are on the same level as the people distributing malware to mine crypto on other peoples' machines. They're exploiting resources that aren't theirs to get rich quick from stupid investors & market hype. We all suffer so they can get a couple bucks. Thanks, AI & braindead investors. This bubble can't pop soon enough, and I hope it takes a whole lot of terrible people down…

[deleted]

Re: Wikipedia is struggling with voracious AI bot crawlers

#93

Post by mrweasel »

karn97 wrote: post #48
guerrilla wrote: post #33

Earlier quoted context omitted.

This is an interesting model in general: free for humans, pay for automation. How do you enforce that though? Captchas sounds like a waste.

Why not just rate limit every user to realistic human rates. You just punish anyone behaving like a bot.

Because, as pointed out in another post about the same problem: Many of these scrappers make one or two requests from one IP and then move on.

Re: Wikipedia is struggling with voracious AI bot crawlers

#94

Post by iJohnDoe »

Not sure why these AI companies need to scrape and crawl. Just seems like a waste when companies like OpenAI have already done this.

Obviously, OpenAI won't share their dataset. It's part of their competitive stance.

I don't have a point or solution. However, it seems wasteful for non-experts to be gathering the same data and reinventing the wheel.

Re: Wikipedia is struggling with voracious AI bot crawlers

#95

Post by scoofy »

guerrilla wrote: post #33
aucisson_masque wrote: post #27

People got to make bots pay. That's the only way to get rid of this world wide DDOSing backed up by multi billions companies. There are captcha to block bots or at least make them pay money to solve them, some people in Linux community also made tools to combat that, i think something that use a little cpu energy. And in the same time, you offer an api, less expensive than the cost to crawl it, and everyone win. Mult…

This is an interesting model in general: free for humans, pay for automation. How do you enforce that though? Captchas sounds like a waste.

Honeypots in JS and CSS

I've been dealing with this over at golfcourse.wiki for the last couple years. It fucking sucks. The good news is that all the idiot scrapers who don't follow robots.txt seem to fall for the honeypots pretty easily.

Make the honeypot disappear with a big CSS file, make another one disappear with a JS file. Humans aren't aware they are there, bots won't avoid them. Programming a bot to look for visible links instead of invisible links is challenging. The problem is these programmers are ubiquitous, and since they are ubiquitous they're not going to be geniuses.

Honeypot -> autoban

Re: Wikipedia is struggling with voracious AI bot crawlers

#96

Post by insane_dreamer »

PeterStuer wrote: post #86
insane_dreamer wrote: post #78

Earlier quoted context omitted.

> the rest of their corruptions the "corruption" accusations are mostly BS and the usual ideological differences I'll take Wikipedia, with all its warts, over $BigTech and $VC-driven (==Ad-driven) companies/orgs any day, and it's not even close.

You don't need a VC if your sitting on $415,9 Million.

They didn't start with $415M, and they didn't take VC money to get there either -- that's the point.

Re: Wikipedia is struggling with voracious AI bot crawlers

#97

Post by milesrout »

voidUpdate wrote: post #9
graemep wrote: post #5

Wikipedia provides dumps. Probably cheaper and easier than crawling it. Given the size of Wikipedia it would be well worth a little extra code. it also avoids the risk of getting blocked, and is more reliable. It suggest to me that people running AI crawlers are throwing resources at the problem with little thought.

Maybe they just vibe-coded the crawlers and that's why they don't work very well or know the best way to do it

We shouldn't use that term. "Vibe coding". Nope. Brainless coding. That is what it is. It's what beginners do: programming without understanding what they--or their programs--are doing. The additional use of a computer program that brainlessly generates brainless code to complement their own brainless code doesn't mean we should call what they are doing by a new name.

Re: Wikipedia is struggling with voracious AI bot crawlers

#98

Post by joquarky »

diggan wrote: post #4

This has to be one of strangest targets to crawl, since they themselves make database dumps available for download ( https://en.wikipedia.org/wiki/Wikipedia:Database_download ) and if that wasn't enough, there are 3rd party dumps as well ( https://library.kiwix.org/#lang=eng&category=wikipedia ) that you could use if the official ones aren't good enough for some reason. Why would you crawl the web interface when the…

> Why would you crawl the web interface when the data is so readily available in a even better format?

Have you seen the lack of experience that is getting through the hiring process lately? It feels like 80% of the people onboarding are only able to code to pre-existing patterns without an ability to think outside the box.

I'm just bitter because I have 25 years of experience and can't even get a damn interview no matter how low I go on salary expectations. I obviously have difficulty in the soft skills department, but companies who need real work to get done reliably used to value technical skills over social skills.

Re: Wikipedia is struggling with voracious AI bot crawlers

#99

Post by joquarky »

marginalia_nu wrote: post #82
jerf wrote: post #42

Earlier quoted context omitted.

To be a "good" web crawler, you have to go beyond "not bad coding". If you just write the natural "fetch page, fetch next page, retry if it fails" loop, notably, missing any sort of wait between fetches, so that you fetch as quickly as possible, you are already a pest. You don't even need multiple threads or machines to be a pest; a single machine on a home connection fetching pages as quickly as it can be already be…

I think most crawlers inevitably tend to turn into spaghetti code because of the number of weird corner cases you need to deal with. Crawlers are also incredibly difficult to test in a comprehensive way. No matter what test scenarios you come up with, there's a hundred more weird cases in the wild. (e.g. there's a world's difference between a server taking a long time to respond to a request, and a server sending hea…

I thrive for these kinds of moving-target challenges. But nobody will hire.

Re: Wikipedia is struggling with voracious AI bot crawlers

#100

Post by johnnyanmac »

jerf wrote: post #87
zokier wrote: post #71

Earlier quoted context omitted.

Identity verification is not that far fetched these days. For europeans you got eIDAS and related tech, some other places have similar stuff, for rest of world you can do video based id checks. There are plenty of providers that handle this, it's pretty commonplace stuff.

That does not generically 100% solve the problem of "is this person a human". That ties things to an identity, but the verification that an identity is actually that human is not solved. Stolen identities, forged identities, faked identities, all still problems, and as soon as the full force of the black market capitalist market in such things is turned on that world, it'll still be a big problem. Also video-based ID…

>That does not generically 100% solve the problem of "is this person a human".

you don't need 100% generic problem solvling. a "good enough solution" will block out 90% of low effort bad actors, and that's a huge relief by itself. That 9% will take some steps and be combatted, and that last 1% will never truly be held at bay.

Post reply on HN